How you can use machine learning to effectively understand key concepts buried in lengthy documents
The challenge
As a consultant working for the big four accounting firm, I often need to quickly work through many lengthy documents (PDFs, Word, PPT etc) to understand key concepts buried somewhere in the paragraphs so that I can “pretend to be intelligent” when catching up with my team members and clients.
Just when I thought my college life of reading through textbooks, lecture notes, sample exam papers is over for good. It quickly became apparent to me that understanding key concepts quickly in preparation for exam is never ending, regardless of which stage of life we are at.
The inspiration
But this actually inspired this idea of using machine learning and domain specific inputs from user to assist and turbocharge our productivity to extract relevant context of the concepts so that we know what to read and skip what is not relevant.
In the subsequent section, I will walk you through two examples of how Second Intelligence has proved to be very useful and shed lights on some surprisingly interesting perspective.
Example 1: Use Second Intelligence to help us understand Metaverse
The concepts of Metaverse has exploded, a phrase I want to better understand after watching Mark Zuckerberg’s presentation on Facebook’s name change.
A quick Google search on “Metaverse” gives me this Wikipedia Page:
The answer is pretty straight forward after a quick read: A Metaverse is a network of 3D virtual worlds focused on social connection. Mission accomplished right? Yes, but wait to hear what Second Intelligence has to say.
We can add this URL to Second Intelligence and asks
- What is the metaverse?
The Second Intelligence will “read” through the content of this URL and trying to understand Metaverse.
You see how the answer is very different to what we got? It says “a Metaverse is the digital future Mark Zuckerbeg is steering us toward”.
The answer from Second Intelligence is dramatically different to ours but both seem to make sense. And I would prefer to hear both sides (both from my own brain after reading through the definition of second intelligence as well as from the Second Intelligence).
Let’s see how our Second intelligence derived at this answer, which is also shown on the right hand side of the workstation.
You see how our Second intelligence has read through the entire Wikipedia page, including the References section, to give us the additional insights and a very interesting perspective, reflecting on how Mr. mark Zuckerberg is committed to building a Metaverse?
Technically speaking, both answers of “a network of 3D virtual worlds focused on social connection” (the one we would have got from reading through the first sentence of the Wikipedia page) and “the digital future Mark Zuckerbeg is steering us toward” (the one Second Intelligence gave us) are correct, but I would love to have the additional perspective which gives me a deeper understanding / another way to look at it.
I hope this demonstrates the value and benefits of using “a second pair of eyes” or your own secretary / virtual assistant to point you towards something that you might be interested to know but might have missed if without any additional input. This could be very valuable either when you are reporting to your supervisor, or when you are doing an open book exam.
Example 2: use Second Intelligence to understand the basics of finance
This example reminds me of my college days when I had to do open book exams, work through sample exam papers and refer back to thousands of textbook pages and many lecture notes to find the relevant explanation of a simple concepts.
For example, I might have the below textbook on Basics of Finance (this is actually the first Google search result of “Basics of finance pdf”) used here for demonstration purpose.
As a student, either when learning the course or preparing for my open book exams, I need to understand key concepts such as:
- “What is financial market” or
- “What is the most important economic problem”.
In this case, I can simply upload this PDF to Second Intelligence and asks the questions as per below:
If you are a student either studying a university course on finance or doing your open book exams, you can leverage on the power of machine learning to quickly find out the relevant segments that explains the particular concepts you are interested in.
In addition to a straight answer provided (column in the middle), the column in the right extracts and highlights the relevant paragraphs in an effort to help you gather some additional context and how the answer was derived at.
By automatically extracting and highlighting the relevant sentences that concisely explains the concept you are trying to understand, this saves you the time to search through your textbooks and lecture notes, so that you can focus your energy on interpreting and understanding the concepts, rather than the manual work of locating the relevant paragraphs from the thousand of irrelevant pages.
I hope this tool turbocharges your productivity in learning key concepts, whether in university, or at work, or trying to understanding an interesting topic.
Additional features
Though this feature appears to be less exciting, I think it is worth mentioning and is a great built-in feature to help you narrow down and locate the relevant segments too.
It essentially allows you to input a few keywords (for example, finance, markets, metaverse etc) and Second Intelligence will extract all paragraphs containing these keywords (including its variations such as the singular and plural forms).
This works very similar to your pdf reader / browser’s search function with two enhanced aspects:
- It searches through multiple datasources of various formats (PDF and Microsoft Word) simultaneously and sort and present the result based on relevance
- The search takes into consideration of multiple (all) search words you specify. It also sorts and presents the result based on relevance. For example, if you input two keywords of “finance” and “market” together, it will sort and prioritize the paragraphs that contains both keywords as it believes this paragraph is more relevant to help you understand your domain-specific knowledge base
The below is a screenshot demonstrates how this works.
The disclaimer
Disclaimer: Second Intelligence is my own product which I built from the grounds up after committing 650 distraction free hours learning web development. So please forgive me because I will be biased towards my own baby.
If you found this concept interesting…
You can access the platform via https://second-intelligence.com/ here. Feel free to try it out with the free version.
If you prefer a video walk through of the features, you can watch this YouTube video I prepared:
Conclusion
If you have read through this far, I really appreciate your interest. I hope you found this project interesting and valuable.
I built this project from the grounds up from the very first line of code, after committing 650 hours learning web development. Here is a big shout out to the pandemic and the horrible Sydney weather, which stopped me from going to the beach during the weekends.
Enjoy! And let’s keep motivating where tomorrow is built today.
