SPMF’s architecture (1) The Algorithm Manager

Posted on 2023-01-16 by Philippe Fournier-Viger

In this new series of blog posts, I will talk about the architecture of the SPMF data mining library, and in particular, I will talk about the AlgorithmManager, which is a key component of SPMF, which manages all the algorithms that are provided in SPMF. I will talk about the key idea behind this module and why it is designed the way it is.

But first, let’s have a look at the overall architecture of SPMF. A picture of the architecture is given below.

Basically, SPMF is a library of algorithms, and there are three types of algorithms: (1) algorithms for preprocessing data, (2) data mining algorithms, and (3) algorithms to visualize data or the output of algorithms, as shown by those three boxes:

The Algorithm Manager is a key module from SPMF that manages the list of all available algorithms offered in SPMF. In particular, it provides the list of all algorithms to the user interface and command line interface of SPMF.

The three main methods (functions) of the Algorithm Manager are illustrated below:

To access the algorithm manager from the Java code, we must write. AlgorithmManager.getInstance() to obtain the instance of the Algorithm Manager. Then, we can call two key methods (functions) of the algorithm manager, which are:

  • getListOfAlgorithmsAsString(): returns the list of all algorithms that are offered in SPMF (as a list of strings),
  • getDescriptionOfAlgorithm(): returns the description of an algorithm that has a given name, which allows to know more about the algorithm and also to run the algorithm.

I will next show you some examples about how to use these two functions, while providing more explanations.

Example 1: Obtaining the list of all algorithms offered in SPMF

First, let me show you an example of how to use the AlgorithmManager to obtain the list of algorithms offered in SPMF. Here I wrote a small Java program:

import java.util.List;
import ca.pfv.spmf.algorithmmanager;

public class Example1{

public static void main(String[] args) throws Exception {
    List<String> list =   AlgorithmManager.getInstance().getListOfAlgorithmsAsString(true, true, true);
    for(String name : list) {
        System.out.println(name);
    }
}

}

By running this program, the list of available algorithms from SPMF will be printed in the console like this:

If you look carefully at this output, you will notice that there are two types of elements in that list: (1) names of algorithms (e.g. “Apriori_association_rules”), and (2) names of categories of algorithms (starting with ” — “). For example, in the category ” — CLUSTERING — “, there are several clustering algorithms such as “BisectingKMeans”, “DBScan”, “Hierarchical_clustering” etc. The algorithms are classified into categories to make it easier for users to look for algorithms.

Another thing that you may notice in the above example, is that the method “getListOfAlgorithmsAsString()” has three Boolean parameters:

getListOfAlgorithmsAsString(true, true, true);

Why? Those Boolean parameters are filters. Setting them to true indicate that we want to list all algorithms from the three types of algorithms (the (1) preprocessing algorithms, (2) the data mining algorithms, and (3) the algorithms for visualizations). If we want to see only the data mining algorithms, we would change as follow:

getListOfAlgorithmsAsString(false, true, false);

Example 2: Obtaining information about a specific algorithm

Now, let me show you a second example, where I will explain how to obtain information about a specific algorithms from SPMF. Here is a simple Java program that calls the AlgorithmManager to get information about the “RuleGrowth” algorithm and print the information to the console:

import java.util.Arrays;
import ca.pfv.spmf.algorithmmanager;

public class Example2{

public static void main(String[] args) throws Exception {
    // / Initialize the algorithm manager
    AlgorithmManager algoManager = AlgorithmManager.getInstance();
    DescriptionOfAlgorithm descriptionOfAlgorithm = algoManager.getDescriptionOfAlgorithm("RuleGrowth");

    System.out.println("Name : " + descriptionOfAlgorithm.getName());
    System.out.println("Category : " + descriptionOfAlgorithm.getAlgorithmCategory());
    System.out.println("Types of input file : " + Arrays.toString(descriptionOfAlgorithm.getInputFileTypes()));
    System.out.println("Types of output file : " + Arrays.toString(descriptionOfAlgorithm.getOutputFileTypes()));
    System.out.println("Types of parameters : " + Arrays.toString(descriptionOfAlgorithm.getParametersDescription()));
    System.out.println("Implementation author : " + descriptionOfAlgorithm.getImplementationAuthorNames());
    System.out.println("URL:  : " + descriptionOfAlgorithm.getURLOfDocumentation());
}

}

The result of running this code is that information about the RuleGrowth algorithm is printed in the console:

Name : RuleGrowth
Category : SEQUENTIAL RULE MINING
Types of input file : [Database of instances, Sequence database, Simple sequence database]
Types of output file : [Patterns, Sequential rules, Frequent sequential rules]
Types of parameters : [[Minsup (%), (e.g. 0.5 or 50%), class java.lang.Double, isOptional = false ], [Minconf (%), (e.g. 0.6 or 60%), class java.lang.Double, isOptional = false ], [Max antecedent size, (e.g. 1 items), class java.lang.Integer, isOptional = true ], [Max consequent size, (e.g. 2 items), class java.lang.Integer, isOptional = true ]]
Implementation author : Philippe Fournier-Viger
URL: : http://www.philippe-fournier-viger.com/spmf/RuleGrowth.php

This output indicates the name of the algorithm, the category that it belongs to, its type of input file and output file, the type of parameters that it takes, who is the implementation author and an URL to the documentation of SPMF for that algorithm.

Now lets me explain in more details about how it works.

When we call the method algoManager.getDescriptionOfAlgorithm(“RuleGrowth“), the Algorithm Manager returns an object of type DescriptionOfAlgorithm. The class DescriptionOfAlgorithm is an abstract class designed to store information about any algorithm. Each algorithm in SPMF must have a subclass of DescriptionOfAlgorithm that provide information about the algorithm.

For example, for the RuleGrowth algorithm, there is a class DescriptionAlgoRuleGrowth that is a subclass of DescriptionOfAlgorithm, which provides information about the RuleGrowth algorithm. If you are curious, here is the code of that class:

Code of DescriptionAlgoRuleGrowth

Each subclass of DescriptionOfAlgorithm must implement a set of methods to provide information about the algorithm. Those methods are:

  • getName(): return the name of the algorithm (e.g. RuleGrowth)
  • getAlgorithmCategory(): return the category of the algorithm (e.g. SEQUENTIAL RULE MINING)
  • getURLOfDocumentation(): return an URL to a webpage describing this algorithm
  • runAlgorithm(): this method is used to call this algorithm (apply it)
  • getParameterDescription(): obtain information about all the parameters of the algorithm. This is provided as a list of object of type DescriptionOfParameter. Basically, for each parameter, we have a name, an example, the type of parameter (e.g. Double) and a Boolean indicating if the parameter is optional (e.g. true) or not.
  • getImplementationAuthorNames(): returns the name(s) of who implemented the algorithm
  • getInputFileTypes(): return the types of input files that this algorithm take as input. It is a list of String from the most general type to the most specific.
  • getOutputFileTypes():return the types of output files that this algorithm take as input. It is a list of String from the most general type to the most specific.

To summarize, here is an illustration of the relationship between the Algorithm Manager and the classes DescriptionOfAlgorithm and DescriptionOfParameter:

Example 3: Running an algorithm

Now let me show you how to use the algorithm manager to run an algorithm from SPMF. Lets look at the following example:


import ca.pfv.spmf.algorithmmanager;

public class Example3{

	public static void main(String[] args) throws Exception {
		AlgorithmManager algoManager = AlgorithmManager.getInstance();
		DescriptionOfAlgorithm descriptionOfAlgorithm = algoManager.getDescriptionOfAlgorithm("PrefixSpan");
		
		String[] parameters = new String[]{"0.4","50","true"};
		String inputFile = "contextPrefixSpan.txt";
		String outputFile = "./output.txt";
		descriptionOfAlgorithm.runAlgorithm(parameters, inputFile, outputFile);
	}
}

This Java program calls the function getDescriptionOfAlgorithm to first obtain the description of the PrefixSpan algorithm. Then, the program call the method runAlgorithm() of that description to execute the algorithm on a file called “contextPrefixSpan.txt” and save the result in a file “output.txt”. Note that to run this example, it is necessary that the file “contextPrefixSpan.txt” is located in the right location on your computer or that you give the full path to the file.

Any algorithms from SPMF can be called in a similar way through the AlgorithmManager.

More about the Algorithm Manager

Now that you know more about the AlgorithmManager and its purpose, let me explain a bit more about the internal design of the AlgorithmManager.

When I implemented this module, I wanted to avoid having a hard coded list of algorithms in the code to make the software easier to maintain. Thus, I have decided that each algorithm in SPMF would instead have a class that describes it, which is a subclass of DescriptionOfAlgorithm. For example, the RuleGrowth algorithm has a class DescriptionAlgoRuleGrowth to describe the RuleGrowth algorithm.

Now, internally, to avoid hard coding the list of all algorithms, the AlgorithmManager scans the package “ca.pfv.spmf.algorithmmanager.descriptions;” to automatically find all the subclasses of DescriptionOfAlgorithm. This allows to automatically find all algorithms that are available in SPMF and make the list of them. The AlgorithmManager can then give this list to the user interface of SPMF, etc.

Thus, if we want to add a new algorithm to SPMF, we just need to create a new subclass of DescriptionOfAlgorithm and put it in the package “ca.pfv.spmf.algorithmmanager.descriptions;” and the Algorithm Manager will automatically detect it, which is very convenient.

If you are curious, this detection is done by the following code in the AlgorithmManager class:

which calls this function:

The above function is somewhat complex because it has to work both when SPMF is called as JAR file or when it is called from the source code. I will not explain this code in more details.

Conclusion

In this blog post, I have given an overview of a key module in SPMF, which is the AlgorithmManager. In upcoming blog posts, I will explain other interesting aspects of the architecture of SPMF. Hope that this has been interesting.

==
Philippe Fournier-Viger is a full professor  and the founder of the open-source data mining software SPMF, offering more than 250data mining algorithms. 

Related posts:

A new survey paper on episode mining!
(video) Mining Frequent Itemsets with the Apriori algorithm
What is Machine Learning?
Posted in Data Mining, Data science, Java, open-source, Pattern Mining | Tagged , , , , , , , , , | 2 Comments

How to call SPMF from Visual Basic .Net (VB)?

Posted on 2023-01-15 by Philippe Fournier-Viger

Today, I will explain how to use SPMF from Visual Basic .Net. Previously, I have explained how to call SPMF from C#, from R, from C++ (on Windows) and from Python.

Requirements

Let me first describe the requirements. It is important to have installed Visual Basic (VB) (for the .NET platform) and Java on your computer.

Second, you should download the spmf.jar file from the SPMF website and put it in the same directory as your VB program.

Third, you should make sure that your Java installation is correct. In particular, you should be able to execute the java command from the command line of your computer. If you type “java -version” in the command line of your computer, you should see the version of Java:

If you see this, then it is OK.

If you do not see this but instead get an error that java.exe is not found, it means that you have not installed Java, or that the PATH to Java is not setup properly on your computer so you cannot use it from the command line. If you are using the Windows operating System and you have installed Java, you need to make sure that java.exe is in the PATH environment variable. On Windows 11, you can fix this problem as follows: 1) Press WINDOWS + R, 2) Run the command “sysdm.cpl“, 3) Click the Advanced system settings tab. 4) Click Environment Variables. 5) In the section System Variables find the PATH environment variable and select it. 6) Click Edit. Add the path to the folder containing java.exe, which will be something like : C:\Program Files\Java\jdk-17.0.1\bin (depending on your version of Java and where you have installed it). 7) Click OK and close all windows. Then, you can open a new command prompt and try running “java -version” again to see if the problem is fixed. If you are using another version of Windows or the Linux operating system, you can find similar steps online about how to setup Java on your computer.

1) Launching the GUI of SPMF from a VB program

Now that I have described the basic requirements, I will first show you how to launch the GUI of SPMF from a VB program. For this, it is very simple. Here I give you the code of a simple VB program that calls the Jar file of SPMF to launch the GUI of SPMF:

Imports System

Module Program
    Sub Main(args As String())
        Process.Start("java", "-jar spmf.jar")
    End Sub
End Module

What this program does? It basically just runs the command
java -jar spmf.jar

Running this Visual Basic program will launch the SPMF user interface as shown below:

2) Executing an algorithm from SPMF from a Visual Basic program

Now, let’s look at something more interesting. How can we run an algorithm from SPMF from VB? We just need to modify the above program a little bit. Let’s say that we want to run the Apriori algorithm on an input file called contextPasquier99.txt (this file is included with SPMF and can be downloaded here).

To do this, we need to first check the documentation of SPMF to see how to run the Apriori algorithm from the command line. The documentation of SPMF is here. How to run Apriori is explained here. We find that we can use this command

java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%

to run Apriori on the file contextPasquier99.txt with the parameter minsup = 40% and to save the result to a file output.txt.

Here is an example program that shows how to do this from Visual Basic:

Imports System

Module Program
    Sub Main(args As String())
        Process.Start("java", "-jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%")
    End Sub
End Module

Then, if we run this program in a folder that contains spmf.jar and contextPasquier99.txt, it will show the following information in the console indicating that the Apriori algorithm was run successfully:

And the program will write the output file output.txt as result:

If we open the file “output.txt”, we can see the content:

Each line of this file is a frequent itemset found by the Apriori algorithm. To understand the input and output file format of Apriori, you can see the documentation of the Apriori algorithm.

If you want to call other algorithms that are offered in SPMF beside Apriori, you can lookup the algorithm that you want to call in the SPMF documentation to see how to run it and then change the above program accordingly.

3) Executing an algorithm from SPMF from a VB program and then reading the output file

Now let me show you another example. I will explain how to call an algorithm from an SPMF and then read the output file from a VB program.

Generally, the output of algorithms from SPMF is a text file (such as in the above example). Thus, to read the output of an SPMF algorithm from VB, you just need to know how to read a text file from a VB program.

For example, I modified the previous VB program to run the Apriori algorithm, wait for the termination, and then read the content of the file “output.txt” that is produced by SPMF to show its content in the console.

This is the modified VB program:

Imports System
Imports System.IO

Module Program
    Sub Main(args As String())

        'Call SPMF To execute the Apriori algorithm'
        Dim psi As New ProcessStartInfo("java", "-jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%")
        Dim p As New Process
        p.StartInfo = psi
        p.Start()
        p.WaitForExit()

        'Read the output file'
        For Each line As String In File.ReadLines("output.txt")
            Console.WriteLine(line)
        Next line
    End Sub
End Module

If we run the program, it will run the Apriori algorithm and then read the output file and write each line of the output file in the console, as expected:

We could further modify this program to do something more meaningful with the content of the output file. But at least, I wanted to show you the basic idea of how to read an output file from SPMF from a VB program.

3) Writing an input file for SPMF from a Visual Basic program, and then running an algorithm from SPMF

Lastly, you can also write the input file that is given to SPMF from a VB program by using code to write a text file.

For example, I will modify the example above to write a new text file called “input.txt” that will contain the following data:

1 2 3 4
2 3 4
2 3 4 5 6
1 2 4 5 6

and then I will call SPMF to execute the Apriori algorithm on that file. Then, the program will read the output file “output.txt” from VB. Here is the code:

Imports System
Imports System.IO

Module Program
    Sub Main(args As String())

        'Write an input file
        Using writer As New System.IO.StreamWriter("input.txt", True)
            writer.WriteLine("1 2 3 4")
            writer.WriteLine("2 3 4")
            writer.WriteLine("2 3 4 5 6")
            writer.WriteLine("1 2 4 5 6")
        End Using

        'Call SPMF To execute the Apriori algorithm'
        Dim psi As New ProcessStartInfo("java", "-jar spmf.jar run Apriori input.txt output.txt 40%")
        Dim p As New Process
        p.StartInfo = psi
        p.Start()
        p.WaitForExit()

        'Read the output file'
        For Each line As String In File.ReadLines("output.txt")
            Console.WriteLine(line)
        Next line
    End Sub
End Module

After running this program, the file “input.txt” is successfully created:

And the content of the output file is shown in the console:

Conclusion

In this blog post, I have shown the basic idea of how to call SPMF from VB by calling SPMF as an external program. It is quite simple. It just require to know how to read/write files in VB. Hope that this information will be useful.

==
Philippe Fournier-Viger is a full professor  and the founder of the open-source data mining software SPMF, offering more than 250 data mining algorithms.

Related posts:

An Introduction to Sequential Pattern Mining
High Utility Itemset Mining with a Taxonomy
Some interesting statistics about SPMF
Posted in Data Mining, Data science, Pattern Mining, spmf | Tagged , , , , , , , , , | Leave a comment

How to call SPMF from a C++ Program (Windows)?

Posted on 2023-01-12 by Philippe Fournier-Viger

I will explain how to call SPMF from a C++ program for the Windows platform. If you are interested by other programming languages, you can check my previous blog posts, where I give examples of how to call SPMF from Python and from C# and from R.

To call SPMF from C++ there are various ways. Here, I will show the most simple way, which is to call SPMF as an external program.

Requirements

As SPMF is a Java software, it is important to first install Java on your computer. Moreover, to compile the C++ program from this blog post, I will use Microsoft Visual Studio. If you are using other compilers for C++, the code might be a little different.

Second, you should download the spmf.jar file from the SPMF website.

Third, you should make sure that your Java installation is correct. In particular, you should be able to execute the java command from the command line (terminal) of your computer because we will use the java command to call SPMF. If you type “java -version” in the command line of your computer, you should see the version of Java:

If you see this, then it is OK.

If you do not see this but instead get an error that java.exe is not found, it means that you have not installed Java, or that the PATH to Java is not setup properly on your computer so you cannot use it from the command line. If you are using the Windows operating System and you have installed Java, you need to make sure that java.exe is in the PATH environment variable. On Windows 11, you can fix this problem as follows: 1) Press WINDOWS + R, 2) Run the command “sysdm.cpl“, 3) Click the Advanced system settings tab. 4) Click Environment Variables. 5) In the section System Variables find the PATH environment variable and select it. 6) Click Edit. Add the path to the folder containing java.exe, which will be something like : C:\Program Files\Java\jdk-17.0.1\bin (depending on your version of Java and where you have installed it). 7) Click OK and close all windows. Then, you can open a new command prompt and try running “java -version” again to see if the problem is fixed. If you are using another version of Windows or the Linux operating system, you can find similar steps online about how to setup Java on your computer.

1) Launching the GUI of SPMF from a C++ program

Now that I have explained the basic requirements, I will first show you how to launch the GUI of SPMF from C++. For this, it is very simple. Here I give you the code of a simple C++ program that calls the Jar file of SPMF to launch the GUI of SPMF.

#include <iostream>
using namespace std;

int main()
{
    // Run the graphical interface of SPMF
    const char* command = "java -jar spmf.jar";
    system(command);
}

What this program does? It basically just runs the command
java -jar spmf.jar

By running this program, SPMF is successfully launched:

spmf data mining interface

2) Executing an algorithm from SPMF from a C++ program

Next, I will explain something more useful, that is how to run an algorithm from SPMF from a C++ program? We will modify the above program to do this. Let’s say that we want to run the Apriori algorithm on an input file called contextPasquier99.txt (this file is included with SPMF and can be downloaded here).

To do this, we need to first check the documentation of SPMF to see how to run the Apriori algorithm from the command line. The documentation of SPMF is here. How to run Apriori is explained in this page of the documentation. We find that we can use this command

java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%

to run Apriori on the file contextPasquier99.txt with the parameter minsup = 40% and to save the result to a file output.txt.

To do this from C++, we can write a simple C++ program like this:

 #include <iostream>
using namespace std;

int main()
{
 
    // Run Apriori on the text file
    const char* command = "java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%";
    system(command);
}

If we execute this program, it will show that this in the console:

And the program will produce the file output.txt as result:

If we open the file “output.txt”, we can see the content:

Each line of this file is a frequent itemset found by the Apriori algorithm. To understand the input and output file format of Apriori, you can see the documentation of the Apriori algorithm.

If you want to call other algorithms that are offered in SPMF besides Apriori, you can lookup the algorithm that you want to call in the SPMF documentation. An example is provided for each algorithm in the SPMF documentation and explanation of how to run it.

3) Executing an algorithm from SPMF from a C++ program and then reading the output file

Now, I will explain how to read the output file produce by SPMF from a C++ program. When running an algorithm of SPMF such as in the previous example, the output is generally a text file. We can easily read an output file from C++ to obtain the content.

For instance, I modified the previous C++ program to read the content of the file “output.txt” that is produced by SPMF to show its content in the console. The new C++ program is below:

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

int main()
{
 
    // Run Apriori on the text file
    const char* command = "java -jar spmf.jar run Apriori input.txt output.txt 40%";
    system(command);

    // Read and display the content of the output file
    fstream outputFile;
    outputFile.open("output.txt", ios::in); //open a file to perform read operation using file object
    if (outputFile.is_open()) {   //checking whether the file is open
        string sa;
        while (getline(outputFile, sa)) { //read data from the file object and put it into a string.
            cout << sa << "\n"; //print the data of the string
        }
        outputFile.close(); //close the file object.
    }

    system("pause");
}

If we execute this C++ program, it will first call the Apriori algorithm from SPMF. Then, the program will read the content of the output file output.txt line by line and display the content in the console:

We could further modify this program to do something more meaningful with the content of the output file such as reading the content in some data structures to do further processing. But at least, I wanted to show you the basic idea of how to read an output file from SPMF from a C++ program.

3) Writing an input file for SPMF from a C++ program, and then running an algorithm from SPMF

Lastly, you can also write the input file that is given to SPMF from a C++ program by using code to write a text file.

For example, I will modify the example above to write a new text file called “input.txt” that will contain the following data:

1 2 3 4
2 3 4
2 3 4 5 6
1 2 4 5 6

and then I will call SPMF to execute the Apriori algorithm on that file. Then, the program will read the output file “output.txt” from C++. Here is the code:

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

int main()
{
    // Write a text file
    fstream inputFile;
    inputFile.open("xyz.txt", ios::out);  // open a file to perform write operation using file object
    if (inputFile.is_open()) //checking whether the file is open
    {
        inputFile << "1 2 3 4\n";   //inserting text
        inputFile << "2 3 4\n";   //inserting text
        inputFile << "2 3 4 5 6\n";   //inserting text
        inputFile << "1 2 4 5 6";   //inserting text
        inputFile.close();    //close the file object
    }

    // Run Apriori on the text file
    const char* command = "java -jar spmf.jar run Apriori input.txt output.txt 40%";
    system(command);

    // Read and display the content of the output file
    fstream outputFile;
    outputFile.open("output.txt", ios::in); //open a file to perform read operation using file object
    if (outputFile.is_open()) {   //checking whether the file is open
        string sa;
        while (getline(outputFile, sa)) { //read data from the file object and put it into a string.
            cout << sa << "\n"; //print the data of the string
        }
        outputFile.close(); //close the file object.
    }

    system("pause");
}

By running this program, the file “input.txt” is successfully created:

And the Apriori algorithm is applied which produces an output file output.txt. Then, the C++ program reads the content of that file and show it in the console:

Conclusion

In this blog post, I have shown the basic idea of how to call SPMF from C++ by calling SPMF as an external program. It is quite simple. It just require to know how to read/write files in C++, and call an external program.

Hope that this has been interesting.

==
Philippe Fournier-Viger is a full professor  and the founder of the open-source data mining software SPMF, offering more than 110 data mining algorithms. If you like this blog, you can tweet about it and/or subscribe to my twitter account @philfv to get notified about new posts.

Related posts:

Introduction to time series mining with SPMF
An introduction to frequent subgraph mining
200,000 visitors on the SPMF website!
Posted in Data Mining, Data science, open-source, spmf | Tagged , , , , , , , | Leave a comment

How to call SPMF from R?

Posted on 2023-01-10 by Philippe Fournier-Viger

In previous blog posts, I have explained how to call SPMF as an external program from Python and how to call SPMF from C#. Today, I will explain how to call SPMF from an R program.

Requirements

Since SPMF is implemented in Java, the first requirement is to make sure that Java is installed on your computer. And of course, you need to also install R on your computer to run R programs.

Second, you should download the spmf.jar file from the SPMF website.

Third, you should make sure that your Java installation is correct. In particular, you should be able to execute the java command from the command line (terminal) of your computer because we will use the java command to call SPMF. If you type “java -version” in the command line of your computer, you should see the version of Java:

If you see this, then it is OK.

If you do not see this but instead get an error that java.exe is not found, it means that you have not installed Java, or that the PATH to Java is not setup properly on your computer so you cannot use it from the command line. If you are using the Windows operating System and you have installed Java, you need to make sure that java.exe is in the PATH environment variable. On Windows 11, you can fix this problem as follows: 1) Press WINDOWS + R, 2) Run the command “sysdm.cpl“, 3) Click the Advanced system settings tab. 4) Click Environment Variables. 5) In the section System Variables find the PATH environment variable and select it. 6) Click Edit. Add the path to the folder containing java.exe, which will be something like : C:\Program Files\Java\jdk-17.0.1\bin (depending on your version of Java and where you have installed it). 7) Click OK and close all windows. Then, you can open a new command prompt and try running “java -version” again to see if the problem is fixed. If you are using another version of Windows or the Linux operating system, you can find similar steps online about how to setup Java on your computer.

1) Launching the GUI of SPMF from R

Now that I have explained the basic requirements, I will first show you how to launch the GUI of SPMF from R. For this, it is very simple. Here I give you the code of a simple R program that calls the Jar file of SPMF to launch the GUI of SPMF.

#Set the working directory to the folder containing spmf.jar
setwd("C:\\Users\\philippe\\Desktop\\")

#Call SPMF as an external program
system("java -jar C:\\Users\\philippe\\Desktop\\spmf.jar")

What this program does? It basically just runs the command
java -jar spmf.jar

By running this program, SPMF is successfully launched:

spmf data mining interface

2) Executing an algorithm from SPMF from a R program

Next, I will explain something more useful, that is how to run an algorithm from SPMF from an R program? We will modify the above program to do this. Let’s say that we want to run the Apriori algorithm on an input file called contextPasquier99.txt (this file is included with SPMF and can be downloaded here).

To do this, we need to first check the documentation of SPMF to see how to run the Apriori algorithm from the command line. The documentation of SPMF is here. How to run Apriori is explained in this page of the documentation. We find that we can use this command

java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%

to run Apriori on the file contextPasquier99.txt with the parameter minsup = 40% and to save the result to a file output.txt.

To do this from R, we can write a simple R program like this:

#Set the working directory to the folder containing spmf.jar and the file contextPasquier99.txt
setwd("C:\\Users\\philippe\\Desktop\\")

#Call SPMF as an external program to run the Apriori algorithm
system("java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%")

Then, it will produce the file output.txt as result in the workind directory:

If we open the file “output.txt”, we can see the content:

Each line of this file is a frequent itemset found by the Apriori algorithm. To understand the input and output file format of Apriori, you can see the documentation of the Apriori algorithm.

If you want to call other algorithms that are offered in SPMF besides Apriori, you can lookup the algorithm that you want to call in the SPMF documentation. An example is provided for each algorithm in the SPMF documentation and explanation of how to run it.

3) Executing an algorithm from SPMF from a R program and then reading the output file

Now, I will explain how to read the output file produce by SPMF from an R program. When running an algorithm of SPMF such as in the previous example, the output is generally a text file. We can easily read an output file from R to obtain the content.

For instance, I modified the previous R program to read the content of the file “output.txt” that is produced by SPMF to show its content in the console. The new R program is below:

#Set the working directory to the folder containing spmf.jar and the file contextPasquier99.txt setwd("C:\\Users\\philippe\\Desktop\\") 

#Call SPMF as an external program to run the Apriori algorithm 
system("java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%")

# Read the output file line by line and print to the console
myCon = file(description = "output.txt", open="r", blocking = TRUE)
repeat{
  pl = readLines(myCon, n = 1) # Read one line from the connection.
  if(identical(pl, character(0))){break} # If the line is empty, exit.
  print(pl) # Otherwise, print and repeat next iteration.
}
close(myCon)
rm(myCon)

If we execute this R program, it will first call the Apriori algorithm from SPMF. Then, the R program will read the content of the output file output.txt line by line and display the content in the console:

We could further modify this program to do something more meaningful with the content of the output file such as reading the content in R data frames to do further processing. But at least, I wanted to show you the basic idea of how to read an output file from SPMF from an R program.

3) Writing an input file for SPMF from a R program, and then running an algorithm from SPMF

Lastly, you can also write the input file that is given to SPMF from a R program by using code to write a text file.

For example, I will modify the example above to write a new text file called “input.txt” that will contain the following data:

1 2 3 4
2 3 4
2 3 4 5 6
1 2 4 5 6

and then I will call SPMF to execute the Apriori algorithm on that file. Then, the program will read the output file “output.txt” from R. Here is the code:

#Set the working directory to the folder containing spmf.jar and the file contextPasquier99.txt setwd("C:\\Users\\philippe\\Desktop\\") 

#  Write an input file for Apriori
file.create("input.txt")
sink("input.txt")                                  
cat("1 2 3 4\r\n")
cat("2 3 4\r\n")
cat("2 3 4 5 6\r\n")
cat("1 2 4 5 6")
sink()

#Call SPMF as an external program to run the Apriori algorithm 
system("java -jar spmf.jar run Apriori input.txt output.txt 40%")

# Read the output file line by line and print to the console
myCon = file(description = "output.txt", open="r", blocking = TRUE)
repeat{
  pl = readLines(myCon, n = 1) # Read one line from the connection.
  if(identical(pl, character(0))){break} # If the line is empty, exit.
  print(pl) # Otherwise, print and repeat next iteration.
}
close(myCon)
rm(myCon)

After running this program, the file “input.txt” is successfully created:

And the content of the output file output.txt is shown in the console:

Conclusion

In this blog post, I have shown the basic idea of how to call SPMF from R by calling SPMF as an external program. It is quite simple. It just require to know how to read/write files in R.

Hope that this has been interesting.

==
Philippe Fournier-Viger is a full professor  and the founder of the open-source data mining software SPMF, offering more than 110 data mining algorithms. If you like this blog, you can tweet about it and/or subscribe to my twitter account @philfv to get notified about new posts.

Related posts:

Finding short high utility itemsets!
On the correctness of the FSMS algorithm for frequent subgraph mining
(Video) Sequence prediction with the CPT and CPT+ Models
Posted in Data Mining, Data science, open-source, spmf | Tagged , , , , , , , , | 6 Comments

How to call SPMF from Python (as an external program)?

Posted on 2023-01-09 by Philippe Fournier-Viger

In this blog post, I will explain how to call the SPMF data mining library from Python. Generally, there are two ways to call SPMF from Python.

The first way is to use an unofficial Python wrapper for SPMF such as SPMF.py, and you can check the website of that wrapper to know how to use it.

The second way that I will explain below is just to call SPMF as an external program directly from a Python program.

Requirements

First, you should make sure that you have installed Python  and Java on your computer.

Second, you should download the spmf.jar file from the SPMF website and put it in the same directory as your Python program.

Third, you should make sure that your Java installation is correct. In particular, you should be able to execute the java command from the command line of your computer. If you type “java -version” in the command line of your computer, you should see the version of Java:

If you see this, then it is OK.

If you do not see this but instead get an error that java.exe is not found, it means that you have not installed Java, or that the PATH to Java is not setup properly on your computer so you cannot use it from the command line. If you are using the Windows operating System and you have installed Java, you need to make sure that java.exe is in the PATH environment variable. On Windows 11, you can fix this problem as follows: 1) Press WINDOWS + R, 2) Run the command “sysdm.cpl“, 3) Click the Advanced system settings tab. 4) Click Environment Variables. 5) In the section System Variables find the PATH environment variable and select it. 6) Click Edit. Add the path to the folder containing java.exe, which will be something like : C:\Program Files\Java\jdk-17.0.1\bin (depending on your version of Java and where you have installed it). 7) Click OK and close all windows. Then, you can open a new command prompt and try running “java -version” again to see if the problem is fixed. If you are using another version of Windows or the Linux operating system, you can find similar steps online about how to setup Java on your computer.

1) Launching the GUI of SPMF from a Python program

Now that I have explained the basic requirements, I will first show you how to launch the GUI of SPMF from Python. For this, it is very simple. Here I give you the code of a simple Python program that calls the Jar file of SPMF to launch the GUI of SPMF:

import os
# Run Apriori
os.system( "java -jar spmf.jar")

What this program does? It basically just runs the command
java -jar spmf.jar

By running this Python program, SPMF is successfully launched:

2) Executing an algorithm from SPMF from a Python program

Now, let’s look at something more interesting. How can we run an algorithm from SPMF from Python? We just need to modify the above program a little bit. Let’s say that we want to run the Apriori algorithm on an input file called contextPasquier99.txt (this file is included with SPMF and can be downloaded here).

To do this, we need to first check the documentation of SPMF to see how to run the Apriori algorithm from the command line. The documentation of SPMF is here. How to run Apriori is explained here. We find that we can use this command

java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%

to run Apriori on the file contextPasquier99.txt with the parameter minsup = 40% and to save the result to a file output.txt.

To do this from Python, we can write a Python program like this:

import os

# Run Apriori
os.system( "java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%")

Then, if we run this program in a folder that contains spmf.jar and contextPasquier99.txt, it will produce the file output.txt as result:

If we open the file “output.txt”, we can see the content:

Each line of this file is a frequent itemset found by the Apriori algorithm. To understand the input and output file format of Apriori, you can see the documentation of the Apriori algorithm.

If you want to call other algorithms that are offered in SPMF beside Apriori, you can lookup the algorithm that you want to call in the SPMF documentation to see how to run it and then change the above program accordingly.

3) Executing an algorithm from SPMF from a Python program and then reading the output file

Now let me show you another example. I will explain how to call an algorithm from an SPMF and then read the output file from a Python program.

Generally, the output of algorithms from SPMF is a text file (such as in the above example). Thus, to read the output of an SPMF algorithm from Python , you just need to know how to read a text file from a Python program.

For example, I modified the previous Python program to read the content of the file “output.txt” that is produced by SPMF to show its content in the console.

This is the modified Python program:

import os

# Run Apriori
os.system( "java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%")

# Read the output file line by line
outFile = open("output.txt",'r', encoding='utf-8')
for string in outFile:
    print(string)
outFile.close()

If we run the program, it will run the Apriori algorithm and then read the output file and write each line of the output file in the console:

We could further modify this program to do something more meaningful with the content of the output file. But at least, I wanted to show you the basic idea of how to read an output file from SPMF from a Python program.

3) Writing the input file for SPMF from a Python program and then executing an algorithm from SPMF

Lastly, you can also write the input file that is given to SPMF from a Python program by using code to write a text file.

For example, I will modify the example above to write a new text file called “input.txt” that will contain the following data:

1 2 3 4
2 3 4
2 3 4 5 6
1 2 4 5 6

and then I will call SPMF to execute the Apriori algorithm on that file. Then, the program will read the output file “output.txt” from Python. Here is the code:

import os

# Write a file
f= open("input.txt","w+")
f.write("1 2 3 4\r\n")
f.write("2 3 4\r\n")
f.write("2 3 4 5 6\r\n")
f.write("1 2 4 5 6")
f.close()

# Run Apriori
os.system( "java -jar spmf.jar run Apriori input.txt output.txt 40%")

# Read the output file line by line
outFile = open("output.txt",'r', encoding='utf-8')
for string in outFile:
    print(string)
outFile.close()

After running this program, the file “input.txt” is successfully created:

And the content of the output file is shown in the console:

Conclusion

In this blog post, I have shown the basic idea of how to call SPMF from Python by calling SPMF as an external program. This is not something very complicated as it is necessary to only know how to read and write files.

I have also written some blog posts about how to call SPMF as an external program from R,  and how to call SPMF from C#.

==
Philippe Fournier-Viger is a full professor  and the founder of the open-source data mining software SPMF, offering more than 250 data mining algorithms. 

Related posts:

IEA AIE 2018 conference (a brief report)
(video) Mining Frequent Itemsets with the Apriori algorithm
How to draw an FP-Tree in Latex? (using TIKZ)
Posted in Data Mining, Data science, open-source, Pattern Mining, spmf | Tagged , , , , , , , , , , , , | 3 Comments

How to call SPMF from C#?

Posted on 2023-01-09 by Philippe Fournier-Viger

This blog post is the first of a series of blog post on using SPMF from different programming languages. Today, I will explain how to call the SPMF data mining library from C#. In other blog posts, I explain how to call SPMF from R and how to call SPMF from Python.

Requirements

First, you should make sure that you have installed C# and Java on your computer.

Second, you should download the spmf.jar file from the SPMF website and put it in the same directory as your C# program.

Third, you should make sure that your Java installation is correct. In particular, you should be able to execute the java command from the command line of your computer. If you type “java -version” in the command line of your computer, you should see the version of Java:

If you see this, then it is OK.

If you do not see this but instead get an error that java.exe is not found, it means that you have not installed Java, or that the PATH to Java is not setup properly on your computer so you cannot use it from the command line. If you are using the Windows operating System and you have installed Java, you need to make sure that java.exe is in the PATH environment variable. On Windows 11, you can fix this problem as follows: 1) Press WINDOWS + R, 2) Run the command “sysdm.cpl“, 3) Click the Advanced system settings tab. 4) Click Environment Variables. 5) In the section System Variables find the PATH environment variable and select it. 6) Click Edit. Add the path to the folder containing java.exe, which will be something like : C:\Program Files\Java\jdk-17.0.1\bin (depending on your version of Java and where you have installed it). 7) Click OK and close all windows. Then, you can open a new command prompt and try running “java -version” again to see if the problem is fixed. If you are using another version of Windows or the Linux operating system, you can find similar steps online about how to setup Java on your computer.

1) Launching the GUI of SPMF from C#

Now that I have explained the basic requirements, I will first show you how to launch the GUI of SPMF from C#. For this, it is very simple. Here I give you the code of a simple C# program that calls the Jar file of SPMF to launch the GUI of SPMF:

using System.Diagnostics;

Process myProcess = new Process();
myProcess.StartInfo.UseShellExecute = false;
myProcess.StartInfo.RedirectStandardOutput = true;
myProcess.StartInfo.FileName = "java";
myProcess.StartInfo.Arguments = "-jar spmf.jar"; // You could also put a path to the jar file like C:\\Users\\philippe\\Desktop\\
myProcess.Start();
myProcess.WaitForExit(); // If you want to wait for the program to terminate

What this program does? It basically just runs the command
java -jar spmf.jar

By running this C# program, SPMF is successfully launched:

2) Executing an algorithm from SPMF from a C# program

Now, let’s look at something more interesting. How can we run an algorithm from SPMF from C#? We just need to modify the above program a little bit. Let’s say that we want to run the Apriori algorithm on an input file called contextPasquier99.txt (this file is included with SPMF and can be downloaded here).

To do this, we need to first check the documentation of SPMF to see how to run the Apriori algorithm from the command line. The documentation of SPMF is here. How to run Apriori is explained here. We find that we can use this command

java -jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%

to run Apriori on the file contextPasquier99.txt with the parameter minsup = 40% and to save the result to a file output.txt.

To do this from C#, we can write a C# program like this:

using System.Diagnostics;

Process myProcess = new Process();
myProcess.StartInfo.UseShellExecute = false;
myProcess.StartInfo.RedirectStandardOutput = true;
myProcess.StartInfo.FileName = "java";
myProcess.StartInfo.Arguments = "-jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%";
myProcess.Start();
myProcess.WaitForExit(); // If you want to wait for the program to terminate

Then, if we run this program in a folder that contains spmf.jar and contextPasquier99.txt, it will produce the file output.txt as result:

If we open the file “output.txt”, we can see the content:

Each line of this file is a frequent itemset found by the Apriori algorithm. To understand the input and output file format of Apriori, you can see the documentation of the Apriori algorithm.

If you want to call other algorithms that are offered in SPMF beside Apriori, you can lookup the algorithm that you want to call in the SPMF documentation to see how to run it and then change the above program accordingly.

3) Executing an algorithm from SPMF from a C# program and then reading the output file

Now let me show you another example. I will explain how to call an algorithm from an SPMF and then read the output file from a C# program.

Generally, the output of algorithms from SPMF is a text file (such as in the above example). Thus, to read the output of an SPMF algorithm from C#, you just need to know how to read a text file from a C# program.

For example, I modified the previous C# program to read the content of the file “output.txt” that is produced by SPMF to show its content in the console.

This is the modified C# program:

using System.Diagnostics;
using System;
using System.IO;

// Call Apriori
Process myProcess = new Process();
myProcess.StartInfo.UseShellExecute = false;
myProcess.StartInfo.RedirectStandardOutput = true;
myProcess.StartInfo.FileName = "java";
myProcess.StartInfo.Arguments = "-jar spmf.jar run Apriori contextPasquier99.txt output.txt 40%";
myProcess.Start();
myProcess.WaitForExit(); // wait for the program to terminate

// Read the output file 
using(StreamReader file = new StreamReader("output.txt")) {  
 string ln;  
  
 while ((ln = file.ReadLine()) != null) {  
  Console.WriteLine(ln);  
 }  
 file.Close();  
}

If we run the program, it will run the Apriori algorithm and then read the output file and write each line of the output file in the console:

We could further modify this program to do something more meaningful with the content of the output file. But at least, I wanted to show you the basic idea of how to read an output file from SPMF from a C# program.

3) Writing an input file for SPMF from a C# program, and then running an algorithm from SPMF

Lastly, you can also write the input file that is given to SPMF from a C# program by using code to write a text file.

For example, I will modify the example above to write a new text file called “input.txt” that will contain the following data:

1 2 3 4
2 3 4
2 3 4 5 6
1 2 4 5 6

and then I will call SPMF to execute the Apriori algorithm on that file. Then, the program will read the output file “output.txt” from C#. Here is the code:

using System.Diagnostics;
using System;
using System.IO;

// Write a file
using (StreamWriter writer = File.CreateText(@"input.txt"))
{
    writer.WriteLine("1 2 3 4");
    writer.WriteLine("2 3 4");
    writer.WriteLine("2 3 4 5 6");
    writer.WriteLine("1 2 4 5 6");
}

// Call Apriori
Process myProcess = new Process();
myProcess.StartInfo.UseShellExecute = false;
myProcess.StartInfo.RedirectStandardOutput = true;
myProcess.StartInfo.FileName = "java";
myProcess.StartInfo.Arguments = "-jar spmf.jar run Apriori input.txt output.txt 40%";
myProcess.Start();
myProcess.WaitForExit(); // wait for the program to terminate

// Read the output file 
using(StreamReader file = new StreamReader("output.txt")) {  
 string ln;  
  
 while ((ln = file.ReadLine()) != null) {  
  Console.WriteLine(ln);  
 }  
 file.Close();  
}

After running this program, the file “input.txt” is successfully created:

And the content of the output file is shown in the console:

Conclusion

In this blog post, I have shown the basic idea of how to call SPMF from C# by calling SPMF as an external program. It is quite simple. It just require to know how to read/write files in C#.

Hope that this has been interesting.

I will later post similar tutorials for other programming languages such as Python, and R so as to make it easier to use SPMF from other languages.

==
Philippe Fournier-Viger is a full professor  and the founder of the open-source data mining software SPMF, offering more than 110 data mining algorithms. If you like this blog, you can tweet about it and/or subscribe to my twitter account @philfv to get notified about new posts.

Related posts:

How to find cost-effective patterns in data?
Brief report about DAWAK 2019 / DEXA 2019
(video) TKQ : Top-K Quantitative High Utility Itemset Mining
Posted in Data Mining, Data science, open-source, spmf | Tagged , , , , , , , , , | 4 Comments

Happy New Year 2023!

Posted on 2023-01-05 by Philippe Fournier-Viger

To all readers of this blog, I would like to wish you a happy new year 2023!

Recently, I have been very busy due to the end of semester and also having COVID. But now, I am fully recovered.

Two things that I would like to talk about.

  1. Recently, I have learnt the sad news that a researcher from the field of data mining has passed away, who was still very young (around 50 years old). He was a good researcher and also a very friendly person that I have met several times at conferences. This reminds me that life can be short and it is important to enjoy it and also take care of your health. As researchers, we often work very hard. But, we should also think about having more balance between work and other aspects of life so as to be more healthy, and also to do sport, eat well and sleep well. I talk more about success and health for researchers in this blog post.
  2. On a different topic, I have recently released a new version of the SPMF data mining software (SPMF version 2.59, which you can download here). It offers two new tools: a graph viewer and an algorithm explorer (which I previously described on this blog), and also three new algorithms for periodic pattern mining, contributed by Prof. Vincent Nofong. I recommend to check out these new algorithms (PPFP, NPFPM and SRPFPM). They offer several possibilities for further research and applications.

This was just a short blog post to wish you a happy new year, talk to you about life, and tell you about the new version of SPMF.

==
Philippe Fournier-Viger is a full professor  and the founder of the open-source data mining software SPMF, offering more than 110 data mining algorithms. If you like this blog, you can tweet about it and/or subscribe to my twitter account @philfv to get notified about new posts.

Related posts:

An Overview of Pattern Mining Techniques
What is a Closed Itemset and Why is it Useful?
SPMF 3.0: Towards even more efficiency
Posted in General, Uncategorized | Tagged , , | Leave a comment

Brief report about the BDA 2022 conference

Posted on 2022-12-20 by Philippe Fournier-Viger

This week, I wasattending the BDA 2022 conference, which is the 10th International Conference on Big Data Analytics. The BDA 2022 conference washeld in Hyderabad, India from the 20th to 22nd December 2022.

The BDA conference is an international conference organized in India, which is quite good and is published by Springer in the Lecture Notes in Computer Sciences series. I have previously attended in 2019 (see my report of BDA 2019). This year, I am attending it again as co-author of a paper and also as a keynote speaker and moderator for a panel.

My report for this conference is a little short because I have been a bit sick during the conference and could not attend all the presentations due to this.

Proceedings

All papers in BDA are published in Springer LNCS which gives good visibility and indexing.

Opening session

I have first attended the opening session. Below are some slides from the opening session that provide some interesting information.

The BDA conference is held every year in different cities in India:

The registration fee is quite reasonable:

About the program of BDA 2022, there was 41 valid submissions, from which 14 were accepted, that is 7 short papers and 7 full papers.

There was several keynotes at the BDA conference. This year, I am one of the keynote speakers. I gave a talk about pattern mining.

There was also several invited talks, some from IBM and the Bank of America, which is quite interesting.

There was also a panel on big data analysis for attaining sustainability. I am one of the two moderators for this panel.

A lot of people are working behind the scene for this conference:

Paper presentations

There was many paper presentations.

My collaborator from Tunisia, Prof. Khaled Belghith presented a paper on using high utility itemsets for transaction embeddings, which may be interesting for those working on pattern mining as it is a kind of bridge between machine learning and pattern mining:


Belghith, K., Fournier-Viger, P., Jawadi, J. (2022). Hui2Vec: Learning Transaction Embedding Through High Utility Itemsets. Proc. of 10th Intern. Conf. on Big Data Analytics (BDA 2022), Springer, to appear.

Panel on data science for sustainable development goals

There was a very interesting panel on data science for sustainable development goals with Prof. Masaru Kitsuregawa (The University of Tokyo), Prof. Longbing Cao (University of Technology of Sydney), Prof. Yun Sing Koh (University of Auckland), and Jaideep Srivastava (University of Minosota).

The panelists brought several interesting perspectives. In particular some cases study was discussed about algae bloom detection and about environmental monitoring. Besides, some other topics were discussed such as the importance of large data centers, health monitoring devices, data collection, and data sovereignty to name a few. The four invited experts also talked about the challenges of interdisciplinary work.

Conclusion

This is a short report about BDA 2022 because I have been sick during the event and I did not attend many activities due to this. But the BDA conference in general is a well-organized conference. The program is good with many excellent guests and speakers, and I also I know several researchers involved in this conference. Thus, I will be looking forward to attending it again.

—
Philippe Fournier-Viger is a professor of Computer Science and also the founder of the open-source data mining software SPMF, offering more than 120 data mining algorithms.

Related posts:

Brief report about the CCF-AI 2021 conference
How to Analyze the Complexity of Pattern Mining Algorithms？
Introduction to frequent subranking mining
Posted in Big data, Conference, Data Mining, Data science | Tagged , , , , | Leave a comment

Unethical reviewers in academia (part 3)

Posted on 2022-12-20 by Philippe Fournier-Viger

Previously, I wrote two blog posts about unethical reviewers in academia (part 1 and part 2). It is not that I like this topic, but today, I will talk again about that. Why? Because, I keep encountering them, unfortunately. It is something very common.

What is an unethical reviewer? As I explained in previous blog posts, there are various types of unethical behaviors that a reviewer may have such as (1) reviewing his own papers, (2) reviewing papers while having some other conflict of interests, or just (3) asking authors to cite his papers to boost his citation count.

Recently, the last case happened again. A collaborator had a paper rejected by two reviewers. And the two reviewers asked to cite between 3 to 5 irrelevant papers. One of the reviewer even put some comments that were unrelated to the paper, which shows that he did not even took the time to do his work seriously, in a hurry to boost his citations. This is some unprofessional behavior and result in wasting time and reduce the quality of the peer-review process.

Personally, every time that this happens, I am a bit angry and because of this phenomenon I think that there are several people in academia that do not care about research and honesty. It is for example, easy to find the profiles of some researchers on Google Scholar who suddenly have thousands of citations but that come from random journals, so it is obvious that they cheat rather than obtaining citations due to the quality of their research work.

So what to do in this situation?

Unfortunately, the balance of power is unequal between authors and reviewers. For the authors who submit a paper to a journal, if the paper is rejected due to unethical reviews, what can he do? He can write an e-mail to the associate-editor or editor-in-chief to complain but from my experience, decisions are almost never reversed in a journal. In fact, I have never seen the option of reversing a decision to even be available in paper management systems from journals. In the best case, maybe the editor could ask to submit the paper again but usually editors are very busy (some journals receive thousands of papers per year!) and I think many editors do not want to take care of authors who argue about the decisions of papers no matter what is the reason.

So what else could be done?

In my opinion, even if has few chances of working, the best is perhaps to send an e-mail to the associate editor and/or editor-in-chief to report the unethical behavior. Maybe that the reviewer could then be blacklisted or that a note could be put in its user profile of the management system as a result. But I would still not bet on this…

In my opinion, if we want something to change about this, the main persons who have power over a journal are the publishers, the societies that are responsible of these journals (e.g. ACM and IEEE), and the companies that take care of impact factors and other academic metrics and rankings of journals.

For example, in a famous case several years ago, an IEEE journal (the IEEE Transactions on Industrial Informatics) lost its impact factor due to citation stacking (artificially increasing the number of self-citations). Losing the impact factor is a serious consequence for the journal that can make things change. So a possibility to make things change is to also complain to the publisher or affiliated societies. This can have some impact although I did not see this happen often.

Another possibility would be to create an online public website where every researcher could upload the potentially unethical reviews that they have received. These reviews could be categorized by journals, and perhaps by authors of papers that reviewers ask to cite. This could show some interesting trends and could perhaps make some things to change. But it would also require to have some moderator to verify such website, and who would take care of this? It would certainly not be a perfect solution and perhaps that people would still find a way to game that system…

Another possibility is to have some external persons that occasionally check what is happening inside the different journals to evaluate them. I think that this is something that does exist. But I do not think that it is for all journals and obviously in some journals nothing is changing over the years.

That is all for today. I just wanted to post my thoughts about this topic once again but this time by discussing also some other solutions.

—
Philippe Fournier-Viger is a computer science professor and founder of the SPMF open-source data mining library, which offers more than 170 algorithms for analyzing data, implemented in Java.

Related posts:

The problem with Short Papers
How to answer reviewers for a journal paper revision?
Why it takes so long for a journal paper to be reviewed?
Posted in Academia | Tagged , , , , | Leave a comment

CFP Special issue on Data Science to Transform How We Work and Live

Posted on 2022-12-11 by Philippe Fournier-Viger

I co-organize a special issue in the DSE (Data Science and Engineering) journal, published by Springer, about how data science can be used to transform how we work and live.

Guest Editors:

  • Yee Ling Boo, RMIT University, Melbourne, Australia
  • Manik Gupta, BirIa Institute of Technology and Science, Pilani (BITS Pilani), Hyderabad, India
  • Weijia Zhang, Southeast University, Nanjing, China
  • Philippe Fournier-Viger, Shenzhen University, Shenzhen, China

The planned schedule is as follows:

Submission Deadline: 30 April 2023
Expected publication: December 2023

See the special issue webpage for the DSE journal for more details:
Data Science and Engineering | Call for papers: special issue on “the innovative use of data science to transform how we work and live” (springer.com)

Related posts:

Why it takes so long for a journal paper to be reviewed?
Why Journal Special Issues are Popular?
How journal paper similarity checking works? (CrossCheck)
Posted in cfp | Tagged , , , , , | Leave a comment