最新消息:Welcome to the puzzle paradise for programmers! Here, a well-designed puzzle awaits you. From code logic puzzles to algorithmic challenges, each level is closely centered on the programmer's expertise and skills. Whether you're a novice programmer or an experienced tech guru, you'll find your own challenges on this site. In the process of solving puzzles, you can not only exercise your thinking skills, but also deepen your understanding and application of programming knowledge. Come to start this puzzle journey full of wisdom and challenges, with many programmers to compete with each other and show your programming wisdom! Translated with DeepL.com (free version)

java 17 - mybatis springboot data copy from 1 databse server to another - Stack Overflow

matteradmin9PV0评论

Using mybatis batch session template to copy data from a table having millions of records to another table in another oracle server.

@Bean(value = "batchSqlSession")
@Autowired
public SqlSessionTemplate batchSqlSession(SqlSessionFactory sqlSessionFactory) {       
    return new SqlSessionTemplate(sqlSessionFactory, ExecutorType.BATCH);
}

Repository Class

@Repository
@Slf4j
public class EmployeeDAO {
    private final SqlSessionTemplate sqlSessionTemplate;
    private final EmployeeMapper employeeSessionMapper;
    public EmployeeDAO (EmployeeMapper employeeSessionMapper,
                          @Qualifier("batchSqlSession") SqlSessionTemplate sqlSessionTemplate) {
        this.sqlSessionTemplate = sqlSessionTemplate;
        this.employeeSessionMapper = sqlSessionTemplate.getMapper(EmployeeMapper.class);
    }

    public List<EmployeeEnt> getEmployeeEnt() {
        return this.employeeSessionMapper.getEmployeeEnt();
    }

    @Transactional
    public void insertEmployeeEnt(List<EmployeeEnt> employeeEntList) {
        employeeEntList.stream().parallel().forEach(employeeEnt-> {
            try{
                employeeSessionMapper.insertEmployeeEnt(employeeEnt);
            } catch(Exception ex){
                log.error("Exception - {} while inserting data {}",ex.getMessage(),employeeEnt.toString());
            }
        });
        sqlSessionTemplate.flushStatements();
        sqlSessionTemplate.clearCache();
    }

    public void truncateEmployeeEnt() {
        this.employeeSessionMapper.truncateEmployeeEnt();
        sqlSessionTemplate.flushStatements();
        sqlSessionTemplate.clearCache();
    }
}

From service class

new EmployeeDAO().truncateEmployeeEnt();
new EmployeeDAO().insertEmployeeEnt(employeeDAO.getEmployeeEnt());

Mapper

<insert id="employeeEnt" parameterType="EmployeeEnt">        
            Insert into EMPLOYEE_DET(..) values {#{empId},#{empName},...
</insert>

getEmployeeEnt() from source Database server is returning 1 Million records and i truncate the target database before inserting the 1 million records. Insert is taking more than 2 hours and it never completes. But if i run this in copying data between 2 schemas of same database server, it takes less than 10 minutes, but if it is 2 different oracle servers, it never completes.

Is there anything i am doing wrong to correct and speed up the inserts? I cannot create database links because source is production data and link creation is not allowed. Exporting the data as CSV and then using sqlldr in java to load is an option?

Post a comment

comment list (0)

  1. No comments so far