In this post, I would like to outline some of the progress that has been made since the last post, or last 6 days and organize the suggestions & ideas for this project, to give a bit more focus to the initial stage, which is crucial.
I would also like to thank Dr. Carson Leung for gathering his entire Database and Data mining lab team for a meeting to discuss my project. I truly appreciate all the suggestions and ideas expressed by Terry Jiang, Richard MacKinnon, Aaron Peddle, Vanessa Reimer and, of course, Dr. Carson himself.
Things to do right now
- As this point, it is crucial to spend time developing an efficient data parser. Regardless of the visualization technique that we choose, it is critical to have fast and efficient storage of data. At least for the processing part, all the data need to be loaded into memory and then processed. In the latter stages, it might be useful to implement level-by-level data loading & processing, once the data sets exceed 500mb in size. However, for now, it is sufficient to develop a efficient storage of data for datasets of at least a million records, which will be the size of the test dataset. Some performance evaluation will follow.
- Explore Unity. At this point, it might sound strange, but actually there is a lot of optimizations in Unity that I won't have neither time nor skills to implement. If, upon exploring, I discover that it is highly efficient at generating and rendering large sets of distinct graphical objects, and provides tools for interaction with them, I will consider switching to Unity. At this point, this is less important than point expressed in 1).
Things to do in the future
- Take the suggestions offered at the before mentioned meeting and put them into effect. My initial impressions are that it is not always necessary to display all the information (as there is just too much of it), instead focus on something that strikes a good balance between clarity and amount of information presented.
- Perhaps, build a quick and easy prototype to evaluate the feel and appeal of the design proposed in the meeting. I will be writing shortly about some designs that were proposed, outlining their pros and cons, giving details and my own thoughts.
- Think about clustering. It is as much as I have on this at the moment.
Overall, there is a lot of work ahead. Next post is planned in a week, and it will be going into more details on various designs and ideas that were discussed during the meeting. Some sketches will be created as well to better document the look of the prototype visualization.