I have used this platform for many years now and as well as having a background as a professional software developer (and general tech wrangler!). I am for this to be constructive and helpful, rather than critical.
OK, where to start! Perhaps a little about why NT8 performance can sometimes suck, and the reasons behind it, not all of which are NinjaTraders fault.
Firstly, and without trying to get too technical, a little about how I understand Ninja is constructed and why this affects performance for some of us. Whilst Ninja is billed as ‘multi threaded’, and it is, its design can introduce a bottleneck depending on how you have it set up. If you don’t know, your computer typically has a processor that can do multiple things at the same time, from a few to many. These are called worker threads, or just ‘threads’. Ninja IS multi-threaded, so, it will update the UI, load some data, and do other stuff in different threads at the same time. The gotcha? By design, it typically does all operations for a single instrument on a single thread. This means that if you have 6 charts open for a single instrument, the data will flow to each, in turn, one by one, in that single thread. Worse – If you have multiple indicators, your data will need to through each of those, too, one, by one. Think of this as going through doors. Each tick has to pass through each door before the next can be processed. This, or the doors that are slow to open and close on the way, is typically what causes the dreaded ‘chart lag’.
Now, there are reasons they built NT this way and they probably seemed like a good idea at the time, but if you are an order flow trader with all the widgets for example and have multiple charts open for the same instrument, like me, you can get into trouble during volatility.
So, let’s cover a few of the options and how they affect you, in terms of performance.
Each time you add an indicator to the chart, you have 1 very important option. The ‘Calculate’ option. This applies to all indicators. Every tick always flows through the bars and all the indicator logic. Add an indicator to the chart with calculate set to ‘OnEachTick’, and the indicator logic will be called for every single tick. Depending on what the indicator does, this can be quite intensive, especially if poorly coded and in volatility when you can have literally hundreds of them per second coming in. For some things, like counting volume, you need this, but for others, like a moving average, you do not. The ‘OnPriceChange’ option calls the indicator logic when the price actually trades at a different price from the last price. This can avoid a few (or a lot of) ticks in the middle and an MA only needs to be updated if the price changes, anyway, so this is more efficient. OnBarUpdate is the most efficient of them all. Perhaps your MA only needs to be updated on bar close, rather than intra-bar. If you run very short time frame charts like 2 tick range bars, for instance, updating the MA on each bar close is probably the best option. This calls the indicator logic only when a new bar is formed, with the values from the bar just closed. This is the most efficient setting of them all.
On PJSIndies, I recently introduced my own intermediate settings. I have been trading instruments with 100 ticks per point, and even OnPriceChange is too inefficient. Blurts of HFT action can cause massive amounts of small price changes in a few ms. I introduced a ‘MinPriceChange’ setting, so indicator logic is only run when the price moves a certain distance. This gives me much more control and increased platform performance through less resource usage. Maybe NinjaTrader will adopt this! It really is a huge improvement.
Hopefully, this gives you an idea of WHY you might get lag in volatility. But, what can you do about it!? There is a semi-useful tool built into NinjaTrader. From the control centre, from the New menu, open the Output window. Once open, right-click it and choose ‘Ninjascript utilisation monitor’. This will give you a very approximate idea of what indicators are consuming most of your resources at that time. Bear in mind, if you have 5 charts with a single same indie on them, that indie will likely be at the top! It does not mean it is broken. Also, you can expect resource-intensive things like order flow to consume more resources than a bar type, for example. These numbers are a rough guide – I don’t find them that accurate. What you should look out for is anything that is massively out of whack with the other numbers. This may give you some clues as to what is bogging down your Ninja.
So, what are a good few things you can do to increase your performance!? These vary, depending on your technical ability and how far you want to go.
The install itself writes a lot of data to log files in real-time as the platform is running. There is no option on Ninja to change these file locations, but I use Windows File junctions to re-locate the folders to a RamDrive, which is 50x faster than even my SSDs. The log and trace folders can safely be moved in this way. This will of course likely be entirely unsupported by NinjaTrader, but I have run this way for years. Some of my indicators write data to a database. This is also hosted on the ram drive. If your code does the same, maybe you can do this too. Running your platform data from an SSD over an HDD is probably going to make a big difference, given NT data is thousands and thousands of small files which are written in real-time.
Ninjatrader also has a single internal database which stores all your trade history etc. This can get quick large, especially over years! If you can stand to lose all your trade history, you can simply reset it, or delete it. NT will re-create it on startup. If not, I have a utility on my website called NT8 cleanup that will let you remove older data, and then compact the file (it is free, but use it at your own risk).
There is a ‘cache’ folder in the db folder (careful, there may be 2 cache folders!). This can get full of old data or even corrupted over time. Delete it occasionally (when your platform is shut down, of course!).
Review all your indicators and carefully check the ‘Calculate’ option on each of them. Perhaps set all the averages you can to OnBarClose only. RSI for example might be OnBarClose, rather than OnEachTick. If you have any indies set for OnEachTick, really think about if you need that. If you are only interested in price, then OnPriceChange is probably more suited.
Whilst you are doing the above, check a few other things. For instance, if you have ‘AutoScale’ set on your MA, did you know this can cause an additional loop through every single bar on the display to find the min and max scale value? The only scale I typically care about is price. If you are the same, turn autoscale off for these indies. Do you need price markers? I personally hate them for most plots. Again, turn them off. Anything you do not need is a waste of processing time.
Drawing objects – Oh, man – These have cost me days and money! A few things. 1, having drawing objects on your charts typically slows down your charts. This is because, with every re-render of the screen, which typically happens several times a second by default, all those drawing objects have to be looped through (if using native NT drawing objects) and any ones in the display range drawn. The indicators that used the easy route, like Draw.ArrowUp, suffer the worse. Drawing these objects natively via the graphics card (in a function called OnRender) tends to be much more efficient if done correctly, but is actually far more complex and probably out of the scope of most tinkerers. You can check the state of play on your chart. Right-click, go to Drawing Tools, then Drawing Objects – If you have 1000’s of them in there, try to do something about it. It will be hurting your performance.
The other thing with drawing objects – They can cause the lag in a massive way, if you get a corrupt one, even if it is off-screen. I believe the reason for this is the constant exceptions internally that can occur when the object is processed (I’ve done some debugging on this!). For instance, something you drew last week, but your chart is only showing the last few days. If you get unexplained lag you don’t normally experience, try going to each individual chart and right-click, drawing tools, and remove all drawing objects. Answer Yes! I personally have had instances where I forgot to do this and suffered for a couple of days with massive intermittent issues, which were all miraculously fixed, just by clearing the drawing objects. This has happened more than once to me, so, it is certainly something to remember, if you do run into issues, and perhaps especially if you have third-party drawing tools.
A trick – If you have a CPU with enough cores, and thus a few threads to spare, you might try this if you are trading the indexes, for example. If you were trading Nasdaq futures, there is the MNQ and the NQ. Now, prices on each are typically at most a tick or 2 different, and generally only in volatility? Do you get where I am going? Some of your price display charts can be NQ, and others, MNQ. Depending on which one you trade and which you track order flow on, if you do, will dictate the mix, but you might be able to get NT to spread the load in this manner and it is something I do.
A hardware consideration. Bearing in mind the threading model, if you have many charts open for a single instrument, and mainly trade one instrument at a time, a processor with fewer cores and more Ghz may provide your NT setup with a bit more go than a slower clock core with more cores. IMO, despite some comments I’ve seen, modern graphics cards appear to be barely taxed by NT, even with 6+ monitors (I typically get about 3% max usage on a modest card with that many)
Charts – A 60-minute bar chart with a few hundred bars will not consume much in the way of resources. A 2 tick unirenko chart, with thousands of bars and creating new ones at a great pace will consume quite a few resources! Especially if you have indies on that chart too. (remember, each indie will have multiple buckets (series) for every single bar!) More bars, more data, more things to update and track on each tick. Try to limit the number of bars on your chart. For things like unirenko, you can set the numbers of bars back to the display in the data series settings, rather than a fixed number of days. For time charts, if you just need today, set this to 0. Do not load more data than you need.
Volume profiles, etc. You might need tick data for these in the short time frame, but for composite profiles, typically 1-minute data resolution is fine. You’ll save a ton of resources and your charts will load a lot quicker. Bear in mind these often still use tick data live, just 1 minute for the historical fill. If your volume profiles allow, merging ticks in high tick instruments can help with performance (if the indie is coded well) and your trading. On NQ, do you need to see .25 increments? Is .50, or even 1 point fine? On the composite, do you need to see every tick level? I don’t. If your volume profile allows, setting the price resolution to a higher step can mean it has to work less in terms of the number of levels and lines it draws, the data it has to maintain, etc. It can also be less data for YOU to process!
Let me finish part 1 of this, by saying, Ninjatrader8 is the best platform available to me, but with great flexibility and a total lack of quality of code filters, you can produce indies that will grind NT to a halt, fairly quickly. It can take some time to figure out the pitfalls of coding for NT8. However, if done correctly, most of the performance issues can be worked around (but it can be challenging!). I do hope they continue to develop and improve the platform. I honestly do not know of anything better for my purposes.
Now, if you get through this and DON’t have lag, you might notice one thing. The chart STILL lags! OK, maybe by a couple of hundred milliseconds, but this is, again, by design. I believe NT typically refreshes the chart at most, once every 250 milliseconds, even if the indies are running their logic much, much faster. This makes sense, to some degree. Now, the Ninja team believe this is perfectly adequate, but my personal preference is for a faster screen update. There have been studies that show the human eye/brain can react amazingly fast – much faster than you can actually ‘see’ something. I know that sounds weird, but it’s correct. The way I scalp and look at very short-term price action, especially in this day and age of HFT etc., I want to see the wiggles. I’ve learned to interpret them, and I cannot see them if the chart only updates a few times a second. If you like, you can get my free price line indicator (on the website) to optionally FORCE an update more frequently. I typically go from 50-100 ms, on my entry chart ONLY. Now, bear in mind, that this is going to introduce even more strain on your workstation, forcing NT to redraw all the indicator plots and bars more times than it typically would out of the box. However, if you watch short-term price action, it can really bring your chart alive!
Good luck – If this helps, please let me know. I might do part 2 🙂
I’m not really concerned with the speed.
Candlesticks take the most time according to the Monitor.
The program is sloooow to load. Slow to bring up different screens but not slow
on displaying the chart.
What I am concerned about is that I wish it worked right.
If you program strategy builder or code editor to work a certain way, it never works right.
Like if I want to do an action on an ema cross. It wont work.
Its so frustrating that most everything doesnt work right.
Thank you Paul for explaining some of the challenges with NT8…i personally have a love hate relationship with the platform. When NT8 was released, the buzz was it’s ability to use multi-core and i don’t recall ever hearing about one core per instrument. So this was interesting news and helps my understanding when the platform stumbles under market stress (volatility). I trade mainly ES and have many ES & MES charts up, mostly orderflow – ladder charts, volume profile – daily, 15m, 30m to monitor VA and POC during each periodicity.
Something i started with NT7 and carried over to NT8 was changing the resource priority to “realtime” under Task Manager>Details and this seems to provide some relief during times of market stress. I am not a programmer and often wondered how the platform is considered when many others are available. I use your PJSpriceline on every chart and it’s made a huge difference in the platform performance along with a SSD and lots of RAM! I checked a few of my charts and i have 100’s of objects drawn …and I’ll look into this, anything to keep the platform from riding the porcelain bus.
There’s some interesting ideas in here Paul, thanks. Spreading out the indicator workload across ES/MES dataseries is a pretty obvious one I missed.
Drawing objects can definitely cause a lot of slowdowns. The area I first really noticed this was in the strategy analyzer during backtests. I had a couple indicators that were heavy on DrawText calls, the backtest would finish (I could see that the strategy had finished processing all days in the test via debug statements in Output window) and then NT would take almost as long as the backtest took to populate the test results to the summary screen. I eventually realized what it was doing was rendering all the drawing objects on the backtest chart during this time, so for indicators like this, I started including a boolean “Backtest” input parameter to turn off the drawing during backtests, resulting in like 25-40% improvements in total testing time for this particular indicator. ( Disclaimer – I am not good at SharpDX, so I probably went the ‘easy route’ for my drawing objects as you mentioned above).
A question for you if you would:
I like to use separate stop & target brackets for multiple entries on the micros. This can sometime lead to having something like 8+ different OCO brackets open. I can’t find where in the NT docs I read it, but I know that there is something like a 200ms expected response time to fully cancel an order and I can definitely see this lag if I ever click the ‘close’ button instead of letting my bracket orders hit, if I hit ‘close’ the platform pretty much freezes up for 1.5-2+ seconds while it cancels all 16+ orders and if the market is busy (FOMC for instance), the brackets will cancel but the command to actually exit the market will time out, leaving me with a position on even though I’ve clicked ‘close’.
Do you have any advice for this sort of multiple order cancellation scenario?
Hi – Thanks for that. Here’s another trick you can use re the drawing objects – Check for ChartControl == null. If you check and see there is no chartcontrol, don’t draw the object. This is what I do to avoid calls to render objects when it’s pointless, but your method for skipping them during backtest is a good one.
I have noticed the order close thing before. On my platform, I have a close feature that instead of closing the pending orders, actually submits a mkt order to close whatever is there, then closes all working orders. It also checks for 0 qty open on any order execution, so if I am short 3 and I manually buy 3 at mkt, it detects the zero qty and it still loops through and closes all working orders. This has worked well for me. I’ve actually left the odd stop or 2 on in the past by accident! I guess there is the possibility of closing orders and still getting filled if you were very close/very laggy, but that has never happened to me in probably many thousands of uses. It gets done what I want first, which is, exit position, close working orders, in that order.
Yeah, that’s a good idea for me I think. Thanks.