Recently, my team has been working on analyzing COVID-19 genome sequences using pattern mining and other AI techniques. We have recently published a paper in the Applied Intelligence journal about this. In this blog post, I will give some brief overview of this. The PDF of thepaper can be found here:
The main idea of the paper is the following. We have obtained genome sequences of different strains of the COVID-19 virus. These genome sequences can be viewed as strings of letters (nucleotides). For example, below is four sequences of nucleotides:
Then, after preprocessing these sequences, it is possible to analyze them using pattern mining algorithms and other artificial intelligence techniques. The main process is the following:
First we prepare the data (step 1), and then we apply different techniques (step 2). First, we have applied itemset mining, sequential pattern miningand sequential rule mining techniques to find patterns that are common to many genome sequences. Some examples of sequential patterns (sequences of nucleotides that appear often) find by the CM-SPAM algorithm are below:
This is just to give an overview. Other types of patterns are discussed in the paper in more details.
Second, we also tested sequence prediction models to see if the nucleotides in genome sequences can be predicted. We compared various models offered in the SPMF data mining software and got results like this:
In general, prediction of genome sequence does not give a high accuracy but still better than a random prediction. We discuss these results in more details in the paper.
Third, we also designed some mutation analysis algorithm to compare different strains of the coronavirus. For example, by comparing two strains, we identified some mutations:
That is just a brief overview of the paper!
There are many possibilities for extensions of this research. In particular, various other pattern mining algorithms and machine learning algorithms could be applied as well. Using the code and data provided above, you can also make your own research on this topic! Besides, the tools presented in this paper can also be applied to other genome sequences beside the COVID-19 virus.
Hope this has been interesting.
— Philippe Fournier-Viger is a professor of Computer Science and also the founder of the open-source data mining software SPMF, offering more than 120 data mining algorithms.
Today, I am happy to announce that a new version of theSPMF open-source data mining software is released (v. 2.44). This is the download page. This new version was made possible due to several contributors.
What is new?
New algorithms: – LTHUI-Miner for mining the locally-trending high utility itemsets (by Yanjun Yang) – MLHUI-Miner for discovering the multi-level high utility itemsets (by Ying Wang) – AER-Miner for mining attribute evolution rules in a dynamic attributed graph (by Ganghuan He) – TSPIN for mining the top-k stable periodic patterns in a transaction database (by Ying Wang, Peng Yang)
New datasets: – Fruithut, Liquor and E-commerce (prepared by Ying Wang)
Some bug fixes: – Fixed a small bug in DFIN (thanks to Nader Aryabarzan) – Fixed a bug in the user interface for TKG (thanks to..) – Fixed a small error in the DB_LHUI.txt example file
Some new tool: – Added a new tool to fix problems in transactions databases with time and utility information
Some new feature: – Added a feature for saving or loading a trained sequence prediction model from file (for the AKOM, DG, TDAG, LZ78, PPM, CPT, and CPT+ algorithms)
The SPMF project is still under active development. The next version will be released in a few weeks as I still have received several algorithms that are waiting to be added to SPMF. The next release will include several new sequential pattern mining algorithms.
Would you like to contribute to SPMF?
The SPMF project is quite successful due to the numerous contributors that are provided some code, reported bugs and also provided many comments. If you are interested to contribute something (code of some algorithms or other things), please leave me an e-mail at philfv8 AT yahoo DOT com.
In this blog post, I will give an introduction to an interesting data miningtask called frequent subgraph mining, which consists of discovering interesting patterns in graphs. This task is important since data is naturally represented as graph in many domains (e.g. social networks, chemical molecules, map of roads in a country). It is thus desirable to analyze graph data to discover interesting, unexpected, and useful patterns, that can be used to understand the data or take decisions.
What is a graph? A bit of theory…
But before discussing the analysis of graphs, I will introduce a few definitions. A graph is a set of vertices and edges, having some labels. Let’s me illustrate this idea with an example. Consider the following graph:
This graph contains fourvertices (depicted as yellow circles). These vertices have labels such as “10” and “11”. These labels provide information about the vertices. For example, imagine that this graph is a chemical molecule. The label 10 and 11 could represent the chemical elements of Hydrogen and Oxygen, respectively. Labels do not need to be unique. In other words, the same labels may be used to describe several vertices in the same graph. For example, if the above graph represents a chemical molecule, the labels “10” and “11” could be used for all vertices representing Oxygen and Hydrogen, respectively.
Now, besides vertices, a graph also contains edges. The edges are the lines between the vertices, here represented by thick black lines. Edges also have some labels. In this example, four labels are used, which are 20, 21, 22 and 23. These labels represents different types of relationships between vertices. Edge labels do not need to be unique.
Types of graphs: connected and disconnected
Many different types of graphs can be found in real-life. Graphs are either connected or disconnected. Let me explain this with an example. Consider the two following graphs:
The graph on the left is said to be a connected graph because by following the edges, it is possible to go from any vertex to any other vertices. For example, imagine that vertices represents cities and that the edges are roads between cities. A connected graph in this context is a graph where it is possible to go from any city to any other cities by following the roads. If a graph is not connected, it is said to be a disconnected graph. For example, the graph on the right is disconnected since Vertex A cannot be reached from the other vertices by following the edges. In the following, we will use the term “graph” to refer to connected graphs. Thus, all the graphs that we will discuss in the following paragraphs will be connected graphs.
Types of graphs: directed and undirected
It is also useful to distinguish between directed and undirected graphs. In an undirected graph, edges are bidirectional, while in a directed graph, the edges can be unidirectional or bidirectional. Let’s illustrate this idea with an example.
The graph on the left is undirected, while the graph on the right is directed. What are some real-life examples of a directed graph? For example, consider a graph where vertices are locations and edges are roads. Some roads can be travelled in both directions while some roads may be travelled in only a single direction (“one-way” roads in a city).
Some data mining algorithms are designed to work only with undirected graphs, directed graphs, or support both.
Analyzing graphs
Now that we have introduced a bit of theory about graphs, what kind of data mining task can we do to analyze graphs? There are many answers to this question. The answer depends on what is our goal but also on the type of graph that we are analyzing (directed/undirected, connected/disconnected, a single graph or many graphs).
In this blog post, I will explain a popular task called frequent subgraph mining. The goal of subgraph mining is to discover interesting subgraph(s) appearing in a set of graphs (a graph database). But how can we judge if a subgraph is interesting? This depends on the application. The interestingness can be defined in various ways. Traditionally, a subgraph has been considered as interesting if it appears multiple times in a set of graphs. In other words, we want to discover subgraphs that are common to multiple graphs. This can be useful for example to find association between chemical elements common to several chemical molecules.
The task of finding frequent subgraphs in a set of graphs is called frequent subgraph mining. As input the user must provide:
a graph database (a set of graphs)
a parameter called the minimum support threshold (minsup).
Then, a frequent subgraph mining algorithm will enumerate as output all frequent subgraphs. A frequent subgraph is a subgraph that appears in at least minsup graphs from a graph database. For example, let’s consider the following graph database containing three graphs:
Now, let’s say that we want to discover all subgraphs that appear in at least three graphs. Thus, we will set the minsup parameter to 3. By applying a frequent subgraph mining algorithm, we will obtain the set of all subgraphs appearing in at least three graphs:
Consider the third subgraph (“Frequent subgraph 3”). This subgraph is frequent and is said to have a support (a frequency) of 3 since it appears in three of the input graphs. These occurrences are highlighted in red, below:
Now a good question is how to set the minsup parameter? In practice, the minsup parameter is generally set by trial and error. If this parameter is set too high, few subgraphs will be found, while if it is set too low, hundred or millions of subgraphs may be found, depending on the input database.
Now, in practice, which tools or algorithms can be used to find frequent subgraphs? There exists various frequent subgraph mining algorithms. Some of the most famous are GASTON, FSG, and GSPAN.
Mining frequent subgraphs in a single graph
Besides discovering graphs common to several graphs, there is also a variation of the problem of frequent subgraph mining that consists of finding all frequent subgraphs in a single graph rather than in a graph database. The idea is almost the same. The goal is also to discover subgraphs that appear frequently or that are interesting. The only difference is how the support (frequency) is calculated. For this variation, the support of a subgraph is the number of times that it appears in the single input graph. For example, consider the following input graph:
This graph contains seven vertices and six edges. If we perform frequent subgraph mining on this single graph by setting the minsup parameter to 2, we can discover the five following frequent subgraphs:
These subgraphs are said to be frequent because they appear at least twice in the input graph. For example, consider “Frequent subgraph 5”. This subgraph has a support of 2 because it has two occurrences in the input graph. Those two occurrences are highlighted below in red and blue, respectively.
Algorithms to discover patterns in a graph database can often be adapted to discover patterns in a single graph.
In this blog post, I have introduced the problem of frequent subgraph mining, which consists of discovering subgraphs appearing frequently in a set of graphs. This data mining problem has been studied for more than 15 years, and many algorithms have been proposed. Some algorithms are exact algorithms (will find the correct answer), while some other are approximate algorithms (do not guarantee to find the correct answer, but may be faster).
Some algorithms are also designed to handle directed or undirected graphs, or mine subgraphs in a single graph or in a graph database, or can do both. Besides, there exists several other variations of the subgraph mining problem such as discovering frequent paths in a graph, or frequent trees in a graph.
Besides, in data mining in general, many other problems are studied related to graphs such as optimization problems, detecting communities in social networks, relational classification, etc.
In general, problems related to graphs are quite complex compared to some other types of data. One of the reason why subgraph mining is difficult is that algorithms typically need to check for “subgraph isomorphisms“, that is to compare subgraphs to determine if they are equivalent. But nonetheless, I think that these problems are quite interesting as there are several research challenges.
I hope that you have enjoyed this blog post. If there is some interest about this topic, I may do another blog post on graph mining in the future.
— Philippe Fournier-Viger is a professor of Computer Science and also the founder of the open-source data mining software SPMF, offering more than 120 data mining algorithms.
In this blog post, I will answer a question that I have received in my e-mail about what is the difference between sequential pattern mining and sequence prediction. I think that this is a good question and sharing the answer can help to clarify some concepts for some people.
Generally speaking, the goal of sequential pattern mining is to find some patterns that appear in many sequencesof symbols. For example, lets say that you have some sequences of purchases made by customers in a retail store. You can then apply a sequential pattern mining algorithm to find sequential patterns, that is to know what are some sequence of purchases that are common to many customers. For example, you may find that <harrypotter1, spiderman, batman> is a sequential pattern. This pattern means that many people have bought the movie Harry potter 1, and then Spiderman, and then Batman. If you find such patterns, it can help you to understand the data. If you are the retail store manager, you may use such pattern to take some business decisions such as to offer some discount to customers on Batman if they previously buy harrypotter and spiderman.
But there are many other usages of sequential patterns. You can also use the sequential patterns to make some sequence prediction. For example, if someone buys Harry Potter 1 and Spiderman, you may predict that he will buy Batman based on the above sequential pattern. This can be used to perform recommendation
Another example about the applications of sequential pattern mining is to find patterns in text documents. A text document is a set of sentences, and each sentences is a sequence of words. Thus, you can apply a sequential pattern mining algorithm to find the sequential patterns that tell you some frequent sequence of words appearing many times in a book. This can tell you about some writing patterns used by some authors, and you can even use these patterns to try to guess who is the author of some anonymous book (if you are curious, I actually did that in a paper: http://www.philippe-fournier-viger.com/FLAIRS2016__AUTHORSHIP_ATTRIBUTION.pdf).
On the other hand, the goal of sequence prediction is to predict what is the next symbol of a sequence of symbols. For example, some people buy the movies Harry Potter 1, Hulk, Batman, and then Star Wars, and we want to know what is the next movie that this person will buy? There are many ways to do sequence prediction. One way is to use the sequential patterns or a variation called sequential rules. For example, we did sequence prediction using sequential rules in apaper to predict the next webpage that someone will click: http://www.philippe-fournier-viger.com/sequential_rules_prediction_2012.pdf But there are also many other models for sequence predictions that do not rely on sequential patterns like the CPT and CPT+ models (video presentation here: https://data-mining.philippe-fournier-viger.com/video-sequence-prediction-with-the-cpt-and-cpt-models/) , the all-k order markov model, the DG model, TDAG, and LZ78.
Thus, to summarize, the goal of sequential pattern mining is to find patterns. You can find these patterns in data for multiple purpose. It can be just to understand the data and learn something about it. It can be to use these patterns to do sequence prediction, or other tasks like clustering, authorship attribution, etc. Thus, sequential pattern mining has many applications and sequence prediction is one of them. And the goal of sequence prediction is to predict the next symbol of a sequence. There are many methods to do sequence prediction and sequential pattern mining is one of them.
Hope that his short answer will be helpful. Some additional blog posts that I wrote on these topics:
— Philippe Fournier-Viger is a professor of Computer Science and also the founder of the open-source data mining software SPMF, offering more than 180 data mining algorithms.
Have you ever received an e-mail from some small conference telling you that they want to invite you to give a talk as a special speaker, honorary speaker or distinguished speaker? I received several e-mails like this and while something it is some valuable invitation, most of the time is just spam. In this blog post, I will discuss this.
Several small conference organizers send unsollicited e-mails to try to attract papers for their conferences and ultimately to collect money in the form of registration fees. As many people ignore these e-mails, some strategies that they use is to mention the title of one of your recent paper and to invite you as a special speaker. This is an example:
Dear Dr. XXXXXX
Please accept my apology if this email bothers you, as I have tried to send you this invitation in last months but without any response from you.
On behalf of the Organizing Committee, it is our delight to extend to NAME_OF_CONFERENCEwhich is going to be held during 2021 (next year) in CITY, Japan.It is our great pleasure and privilege to welcome you to join the NAME_OF_CONFERENCE act as the chair/speaker while presenting about TITLE OF_MY_PAPER.
Another example:
Dear Dr. XXXXX,
Hope you receive this letter in a wonderful mood. Please accept my apology if this email bothers you, as I have tried to send you this invitation to you but without any response. Would you please send a reply?
Thanks for your time to this email from our committee, the committee of XXXXXX-2022 cordially welcome you to share a presentation as a session speaker/chair regarding your research. Sincerely wish you can give us an opportunity to include your research in our program and proceeding.
If you read such e-mails for the first time, you may think that the sender has read your paper and is really interested in your research and wants to invite you as a special guest to their conference to give a talk. So you may be tempted to accept. However, most of the time, this is just SPAM and they only mention the title of one of your paper and that you are invited as a special speaker to catch your attention.
Before accepting such invitation, you should check: (1) is this a well-known conference in your field that has been held for several years? (2) do you know other people having attended it? (3) is it associated with a famous institutions or publisher (beware that the website may be fake and provide misleading information, though). But perhaps the most important is to check if the conference organizers will be asking you to pay a registration fee or give you some special benefits. The reason is that as an invited guest, you would expect some kind of preferential treatment over regular attendees. If they do not give you any special benefits as a special speaker, then you are just another speaker for their conference, and the goal is just to earn money. If you are not sure about whether a conference is legit or not, it is best also to ask your supervisor or senior researchers for their opinion.
There are also many other types of SPAM e-mails that target academic researchers such as e-mails asking you to publish your thesis as a book. I may talk about this in more details in a future blog post! That is all for today!
This is a short blog post tto let you know that I am co-organizing two special issues in some journals related to data mining, data science and machine learning:
(2) Special issue on Generative Adversarial Networks for Multi-Modal Multimedia Computing Journal: Wireless Communication and Mobile Computing Publisher: Wiley / Hindawi Deadline: 28th May 2021 Link: https://www.hindawi.com/journals/wcmc/si/710287/
Note that those are open-access journal. Thus, there is a publication fee. If you do not want to have a publication fee, you may consider submitting your paper to the Data Science and Pattern Recognition journal (DSPR), which is currently free to publish and has a fast review time. I am editor-in-chief of that journal.
If you have any questions or are not sure whether your research is relevant to these journals, you may send me a message, and we can discuss about it.
In this blog post, I will talk about the concept of atomic habits and how it can help you achieve your goals in general, but especially to become a better researcher.
The concept of Atomic Habits, was popularized in a book called “Atomic Habits” by James Clear. The idea in this book is quite simple but yet, it is a powerful way of achieving your goals by applying some simple steps in your daily life.
Setting your goal
The first step is to set some clear goal about what you want to do. This is important as it is the target that you want to achieve. A goal could for example be to improve your paper writing skills.
The importance of the process or system
Having a goal is good but it is yet not enough because persons who succeed and fail still have the same goal (e.g. the winner of a race vs the losers, or the researchers who got a tenured positions vs those who don’t). Thus, what makes the difference between succeeding and failing is not the goal but the system or process that is used to achieve it.
The system of atomic habits
The main idea in the book of James Clear is that we can achieve big goals by changing our daily habits. This can be by adopting some good habits. For example, if your goal is to improve your English writing skills, working on it 20 minutes every day may not do much in the short term but in the long term may lead to major improvements. But it can also be useful to remove the bad habits. For example, one may want to stop wasting too much time browsing the Web every day.
However, as many people knows, it is often hard to start a good habit and keep it for a long time. Many people will for example start to do some physical exercises for a few weeks and then give up quickly. It is also hard to stop bad habits.
To help change habits, the key points proposed by James Clear are:
Make it easy: Do not try to make some changes that are too challenging early. For example, if you decide to study English 5 hours per day, it would be perhaps be difficult to sustain over the long term. It is perhaps better to start with 20-30 minutes per day, and later you may increase. But at first, consistency is what is important and will help you to not give up.
Make it obvious: To make sure that you continue your good habit every day, it is important that you do not forget about it. Thus, you may try to connect your new habit with your previous habits. For example, if you want to take some medicine every day, you may put it beside your toothbrush to not forget to take it. If you want to read a book every night, you may put the book on your pillow.
Make it attractiveand satisfying: Because the long term goals may take a lot of time to achieve, it is important to make sure you associate some short term rewards to your good habits . Thus, you may think about some rewards such as: If I study English every day for one week, I will buy myself a hot chocolate cup.
Make it harder to keep the bad habits: You may think about strategies to make your bad habits harder to do. For example, if you want to drink less alchool, you may put the bottles out of sight.
Track your habits: You may use some book to keep a record of your habits over time.
Find the right environment and the right people. The environment and the people that we interact with also play a role in changing habits. Changing the environment or getting along with other people having the same goals may help.
These ideas are quite simple and can be applied to many aspects of life (loosing weight, etc.) but can be also used by researchers to become better researchers. Some good habits for researchers may be to waste less time on the Internet, to have a fixed schedule and sleep well every day, to eat well, to exercise, to improve writing skills, to write more papers, to write a book, to improve programming skills, to write a blog post every week, to improve presentation skills, etc.
Conclusion
In this blog post, I gave a short overview of the book Atomic Habits and discussed a little bit about how it can be used in the life of a researcher. If you do not have time to read the book, you may have got the main idea from this blog post. There is also some good video presentations by James Clear that can be watched online (it is shorter than reading the book, if you are busy like me!).
If you have any comments, please post in the comment section below.
Today, I will talk about at topic that not many researchers talk about, which is the long and hard road to become a professor in academia, and why some people give up before reaching their goal at different stages of their career. I will talk about this topic because I was recently reading about researchers who decided to leave academia to do something else due to the difficulty of getting a tenured professor position or permanent researcher position. A list of researchers that have posted about the reasons why they have left academia can be found below, and has inspired this blog post: https://docs.google.com/spreadsheets/u/0/d/1OODoiZKeAtiGiI3IAONCspryCHWo5Yw9xkQzkRntuMU/edit
By reading these posts, some observations are:
Several people complained that there are not enough professor positions that are available. This is true as there are much more persons who obtain a PhD than persons who can become a professor.
Someone can become a post-doctoral researcher after the PhD to gain more experience, but there is typically a limit on the number of years that one can work as a post-doctoral researcher. So this is a temporary solution to get more time before finding a professor position. Some persons have done up to 6 years as a post-doctoral researcher after their Ph.D but could still not find a faculty position and thus decided to give up and do something else.
Some persons complained about the low salary of working as a post-doctoral researcher compared to what they would earn in the industry. Some also mentioned the unability to have a stable job, sometimes having to sign a 1 year contract, while some other have been more lucky to sign for 2 or 3 years.
Several people complained about not choosing where they would live (for example having to accept a post-doc position in another city far from their family)
Some people enjoyed working as a post-doctoral researcher but it is is a temporary position as there is typically a limit on the number of years that one can be a post-doctoral researcher.
Some persons complained that many entry-level faculty positions in universities are short-term contracts and are not permanent. This is a reality in many places. In fact, after my Ph.D and doing my post-doc, I even started with a 9 month contract as an adjunct professor position in Canada, before getting a 3 year contract, and now a 5 year contract. Also, several permanent professor who retired were replaced by temporary jobs to reduce costs.
Some people have given up on academia to work in the industry or start their own business among other things. Some have decided to do something completely different such as starting a knitting store! Some have decided to even go back to studying to obtain a degree in another field.
Some people have said that they actually gave up on many other things to try to succeed in academia such as spending less time with their daugther, working every evening and week-ends to work on papers and books, and also gave up on other things that they like. Thus, by leaving academia, some have said that they are now happy to pursue other dreams.
Several adjunct professors have tried hard to get a tenured position but gave up. Some reasons are the inability to get national funding, after applying multiple times and failing. In some countries like USA, the success rate appears to be very low in some fields.
Some researchers have complained about some toxic working environments such as other people trying to sabotage their research, etc.
Some researchers have talked about the negative psychological effect and depression due to various factors such as having to work hard, a toxic work environment, and to pressure to obtain grants and get tenured.
Some people claim that a good amount of luck is required to be successful in academia.
I have to say that it is true that it is not easy to succeed in academia. I personally had to go through many challenges to become a professor and eventually get a full professor position, and be successful in my field. I also had to give up on several other things that I like to succeed. And I had to work hard for more than a decade, almost every day from morning this late at night (which I still do, by the way). But now, I have a good position and I am quite happy of what I do. For me, all this hard work was worth it as I like doing research and Ienjoy teaching. But I certainly gave up on some other things that I enjoy to focus on my research career. For example, I also enjoy other things like learning languages, drawing, playing guitar and running as hobbies, which I do not have too much time to do.
For the young researchers, my first advice is to learn to know yourselfand what you really like. If your dream is to become a professor, it is possible but you need to work hard and work smart to be the most effective and reach your goal. You also need to make a realistic plan of how to attain your goal. I have written several blog posts to give advices about how to be successful in academia that you can read:
Recently, I gave an invited talk at University of Pisa in Italy (online). A PhD student asked me: If you would do a PhD again, what would you do differently? In this blog post, I will answer this question, which I think can be interesting for graduate students.
First, I think that one of the key aspects to consider for a PhD is to choose a good research team, preferably in a good university, where you will have a good research environment and can work on some important research topic.. In my case, I did my PhD in a university that is maybe not so high in the world rankings but is still good, and more importantly my supervisor was great and gave me several opportunities through his social network. Thus, for this, I would not change.
A second important aspect is about time management. If I would do a PhD again, I would try to manage my time in a better way to be more effective. As a student, I had a lot of time but sometimes spent time on things that were not so important. It is important to be able to assess what is the most important and to choose carefully how to spend time. For example, if you have one day left to submit a paper, is it more important to spent it improving the colors of figures or proofreading? Generally, the latter is more useful.
A third important aspect is to collaborate more with other researchers. As a PhD researcher, it is easy to work by yourself on your thesis. But having the feedback of others can be highly valuable. Moreover, collaborating with others can help write more papers and find other opportunities. On this aspect, I did well during my PhD as I had several collaborations but I could have perhaps discussed more about my project with other researchers.
A fourth important aspect is to choose a research topic that you like. Personally, during my PhD, I first started doing something on e-learning before gradually moving towards data mining, which is my current research area. If I had made that decision earlier, it would have been better. But this is easy to say, afterwards. Although I also liked working on e-learning, the community was quite small to work on Intelligent Tutoring Systems and thus it was hard to have some impact in this field despite doing good research. Another reason why I stopped working on this is that conducting experiments was quite time-consuming and complex, while in data mining it can be as simple as running algorithms on a benchmark dataset that you download to test a new algorithm. Besides, I personally like research on algorithm design.
A fifth important aspect is to have clear goals for your career path after the PhD. It is never too early to search for jobs or opportunities such as postdoc positions. I think I did quite well on this part as I got a postdoc position in a good data mining team. But I could have started searching earlier.
A sixth important aspect is to focus on having quality papers in good journals and conferences, recognized worldwide if you intend to have an international career. In some countries like Canada, some conference papers are well regarded in computer science, and there is not much pressure to write journal papers for PhD students. Even, some PhD students may graduate without any papers at some universities. But internationally, several countries consider journal papers as highly important and have various ranking systems to evaluate journals and conferences. For researchers on intend to work internationally, it is thus something important to consider. Sometimes, it is better a few very good papers than have too many papers.
Conclusion
In this blog post, I gave some answers to the question of what would I do differently if I would do another PhD. Hope it was interesting. If you have some comments, please write below.
—
Philippe Fournier-Viger is a full professor working in China and founder of the SPMF open source data mining software.
In this blog post, I will talk briefly about how tweets collected on Twitter can be analyzed to understand the public opinion about COVID-19. This is based on the below research paper, that I have recently participated to:
I will give an overview of the above paper. For more details, you can click on the above link to see the whole research paper.
Why analyzing Tweets? There has been a lot of research about analyzing tweets in the past such as to detect the sentiment and feelings of people on different topics, or even to detect fake news and bots among other things. The interest of analyzing Twitter data is that Twitter is used by millions of people and that tweets are posted in real-time. Thus, tweets can be used to analyze what people are saying about a topic such as the coronavirus.
How can we understand public opinion about COVID-19 on Twitter? In the above research paper, we applied the following methodology. We have first collected thousands of tweets in English about COVID-19 during the first months of the pandemic. Then we applied some clustering algorithms to discover the main themes that were talked about on Twitter related to COVID-19. Moreover, we applied sequential pattern mining algorithms to find frequent words patterns in Tweets.
What have we discovered? We have found several interesting things. For the cluster analysis, we found seven main clusters of tweets that indicate some main themes discussed by Twitter users:
Cluster 1 (green): public sentiments about COVID-19 in the USA.
Cluster 2 (red): public sentiments about COVID-19 in Italy and Iran and a
vaccine,
Cluster 3 (purple): public sentiments about doomsday and science credibility.
Cluster 4 (blue): public sentiments about COVID-19 in India.
Cluster 5 (yellow): public sentiments about COVID-19’s emergence.
Cluster 6 (light blue): public sentiments about COVID-19 in the Philippines.
Cluster 7 (orange): Public sentiments about COVID-19 US Intelligence Report.
For example, this is the cluster 1:
And this is the cluster 2:
Cluster 3:
Some part of cluster 4:
Some part of cluster 5:
Some part of cluster 6:
We also found several patterns related for example to “Coronavirus, testing, lockdown”. Here is for example, some of the most frequent words:
More results are presented in the paper.
The above results represent what the sampled tweets have been talking about on Twitter in English from January to March 2020, related to COVID-19.
Conclusion
In this blog post, I have just given a very brief overview of what can be learnt from Tweets related to public opinion. For more details, please check the above paper! There is also obviously some limitations to that study such that Tweets were not geolocalized and that only the English language was used. If you have any comments you may post in the comment section below. Hope this has been interesting.
