Trusted by over 1.6 million members since 2014 — why not join them?
Best program/way to datamine json with a lot of data (70mb currently)
- Author: Thread starter mreeotw
- Start date
-
- Tags
- dataedit help reqest
For bigger files, I can recommend Editpad Lite or, if you can afford it, the Pro version. Notepad++ unfortunately gets slow, laggy and stuck with big files, therefore I had to discard it. I usually use Editpad when I'm dealing with large database files (up to 4-5GB) without any particular issues, so it should also be good for JSON files as well.
^ Ultraedit is great for large text files. LogExpert is even better (for Windows) if you're talking >500mb. I've used it for multi-gigabyte forum SQL backups.
Though presumably, by the sounds of it, you want to process the data. In which case, either find a language with a package for streaming in huge JSON files and parsing them or find a language with fgets() or equivalent (C, C++, PHP, Perl etc.) and read it in line-by-line to do the parsing yourself. The latter is what text editors like LogExpert do to handle huge files (which is why you should avoid any huge, resource-hungry editors based on Electron/CEF for tasks like this). Frankly, though, 70MB really isn't big by today's standards, so most languages have stdlib facilities or imports that can handle reading the whole thing into RAM for maximum efficiency -- as long as you've got an okay-ish computer. Languages with inferior implementations that copy-by-value everywhere, like PHP, though, will absolutely choke themselves to death creating a many-dimensional associative array of key-value pairs from a 70MB JSON file, so use Perl or Python, which are intended to handle it (obviously C++, Rust etc. will do a better job, but if you're asking this question, you're probably not writing an efficient JSON parser in C++).
Though presumably, by the sounds of it, you want to process the data. In which case, either find a language with a package for streaming in huge JSON files and parsing them or find a language with fgets() or equivalent (C, C++, PHP, Perl etc.) and read it in line-by-line to do the parsing yourself. The latter is what text editors like LogExpert do to handle huge files (which is why you should avoid any huge, resource-hungry editors based on Electron/CEF for tasks like this). Frankly, though, 70MB really isn't big by today's standards, so most languages have stdlib facilities or imports that can handle reading the whole thing into RAM for maximum efficiency -- as long as you've got an okay-ish computer. Languages with inferior implementations that copy-by-value everywhere, like PHP, though, will absolutely choke themselves to death creating a many-dimensional associative array of key-value pairs from a 70MB JSON file, so use Perl or Python, which are intended to handle it (obviously C++, Rust etc. will do a better job, but if you're asking this question, you're probably not writing an efficient JSON parser in C++).
Last edited:
Oops. Just saw the OP date. It's too late to edit my post but you've almost certainly solved it or given up by now. Oh well, maybe it'll help someone. My only other piece of advice is that writing a reliable, generic JSON parser (ironically worse than XML, IMO) from scratch is intermediate to advanced level, depending on the language (Perl = mid, C++ [esp. pre-11/14/17] = hard), so use a verified or at least road-tested package from someone else. I wrote a JSON parser in C++ for the Discogs API and it was hacky as ****. It was sort of halfway between a finite state machine and a spaghetti mess of static callback functions. It worked reliably but only by knowing the exact format and order of Discogs' web service responses. If they ever change it, it'll stop working. It was a hackjob for a highly-specific use case.
Last edited:
