Spring Integration – Polling file creation and modification

Integration, Spring, Spring-Boot · 19 Jul 2016

File support is another of Spring Integration’s endpoints to communicate with external systems. In this case, it provides several components to read, write and transform files. During this post, we are going to write an application which monitors a directory in order to read all files in there. In concrete it does the following:

When the application starts, it reads all files present in the directory.
The application will then keep an eye on the directory to detect new files and existing files which have been modified.

The source code can be found in Github.

1 Configuration

The application is built with Spring Boot, since it eases configuration significantly. To create the initial infrastructure of the application, you can go to https://start.spring.io/, select the Integration module and generate the project. Then you can open the zip file in your favourite IDE.

I added a couple of dependencies to the pom.xml like commons.io or Spring Integration Java DSL. My pom.xml file looks like as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>xpadro.spring.integration</groupId>
  <artifactId>file-read-directory</artifactId>
  <version>0.0.1-SNAPSHOT</version>
  <packaging>jar</packaging>

  <name>file-read-directory</name>
  <description>Demo project for Spring Boot</description>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.3.5.RELEASE</version>
    <relativePath/> <!-- lookup parent from repository -->
  </parent>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <java.version>1.8</java.version>
  </properties>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-integration</artifactId>
    </dependency>

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>

    <!-- Spring Integration - Java DSL -->
    <dependency>
      <groupId>org.springframework.integration</groupId>
      <artifactId>spring-integration-java-dsl</artifactId>
      <version>1.0.0.RELEASE</version>
    </dependency>

    <dependency>
      <groupId>commons-io</groupId>
      <artifactId>commons-io</artifactId>
      <version>2.5</version>
    </dependency>

  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
    </plugins>
  </build>

</project>

The starting point is FileReadDirectoryApplication:

@SpringBootApplication
public class FileReadDirectoryApplication {

    public static void main(String[] args) throws IOException, InterruptedException {
		SpringApplication.run(FileReadDirectoryApplication.class, args);
	}
}

Starting from here, we are going to add the Spring Integration components for reading from a specific folder of the filesystem.

2 Adding the adapter

In order to read from the file system, we need an inbound channel adapter. The adapter is a file reading message source, which is responsible of polling the file system directory for files and create a message from each file it finds.

@Bean
@InboundChannelAdapter(value = "fileInputChannel", poller = @Poller(fixedDelay = "1000"))
public MessageSource<File> fileReadingMessageSource() {
    CompositeFileListFilter<File> filters = new CompositeFileListFilter<>();
    filters.addFilter(new SimplePatternFileListFilter("*.txt"));
    filters.addFilter(new LastModifiedFileFilter());
    
    FileReadingMessageSource source = new FileReadingMessageSource();
    source.setAutoCreateDirectory(true);
    source.setDirectory(new File(DIRECTORY));
    source.setFilter(filters);
    
    return source;
}

We can prevent some types of files from being polled by setting a list of filters to the message source. For this example two filters have been included:

SimplePatternFileListFilter: Filter provided by Spring. Only files with the specified extension will be polled. In this case, only text files will be accepted.
LastModifiedFileFilter: Custom filter. This filter keeps track of already polled files and will filter out files not modified since the last time it was tracked.

3 Processing the files

For each polled file, we will transform its content to String before passing it to the processor. For this purpose, Spring already provides a component:

@Bean
public FileToStringTransformer fileToStringTransformer() {
    return new FileToStringTransformer();
}

Hence, instead of receiving a Message, the processor will receive a Message. The file processor is our custom component which will do something as advanced as printing the file content:

public class FileProcessor {
    private static final String HEADER_FILE_NAME = "file_name";
    private static final String MSG = "%s received. Content: %s";
    
    public void process(Message<String> msg) {
        String fileName = (String) msg.getHeaders().get(HEADER_FILE_NAME);
        String content = msg.getPayload();
        
        System.out.println(String.format(MSG, fileName, content));
    }
}

4 Building the flow

Now that we have all the required components in place, let’s build the flow. We are using Spring Integration Java DSL, since it makes the flow more readable:

@Bean
public IntegrationFlow processFileFlow() {
    return IntegrationFlows
        .from("fileInputChannel")
        .transform(fileToStringTransformer())
        .handle("fileProcessor", "process").get();
    }
    
    @Bean
    public MessageChannel fileInputChannel() {
        return new DirectChannel();
    }

5 Running the application

In my directory, I already have a file called ‘previousFile.txt’. After starting the application, we will create two files and modify one of them.

public static void main(String[] args) throws IOException, InterruptedException {
    SpringApplication.run(FileReadDirectoryApplication.class, args);
    createFiles();
}

private static void createFiles() throws IOException, InterruptedException {
    createFile("file1.txt", "content");
    createFile("file2.txt", "another file");
    appendFile("file1.txt", " modified");
}

If we run the application, we should see the following print statements:

previousFile.txt received. Content: previous content
file1.txt received. Content: content
file2.txt received. Content: another file
file1.txt received. Content: content modified

6 Conclusion

This example shows how simple it is to read files from a directory using Spring Integration, obviously with the help of Spring Boot to simplify the configuration. Depending on your needs, you can add your own custom filters to the message source, or use another one of the provided by Spring, like the RegexPatternFileListFilter. You can check for other implementations here.

If you found this post useful, please share it or star my repository 🙂

I’m publishing my new posts on Google plus and Twitter. Follow me if you want to be updated with new content.