how to config actual-data-nodes while my actual-data-nodes tables is dynamic。 #16725

wlklnn · 2022-04-11T05:46:13Z

My shard-table will be created everyday。 It can't be configured by
<sharding:table-rule logic-table="t_mt" table-strategy-ref="self_hint_shard" actual-data-nodes="——"/>
my maven dependency is

org.apache.shardingsphere
shardingsphere-jdbc-core-spring-namespace
5.1.0

wlklnn · 2022-04-11T07:31:30Z

if the actual-data-nodes was set as not exists table,there is a NPE.

linghengqian · 2022-04-11T07:40:25Z

This answer in Apache ShardingSphere 5.1.0, the best answer is obviously to rely on Zookeeper/Etcd, in Apache ShardingSphere Proxy to dynamically change through DistSQL.
Of course, you can do some things via CI/CD without using a Proxy, because ActualDatanode is actually a Groovy expression that can call static variables and methods of Java classes. If you know Gitlab Runner, I'm sure you know what I'm talking about, there are many applications for this.
Second, ShardingSphere seems to have an agenda to create tables that don't exist at startup. It is pointless to configure a table that does not exist in the ShardingSphere, which will cause metadata checking to fail.

wlklnn · 2022-04-11T08:52:45Z

The actualDatanode is dynamic ,for example, today's ActualDatanode is [table_name_20220401..table_name_20220411], and become [table_name_20220402..table_name_20220412] tomorrow,should i use a Scheduling to set ActualDatanode everyday ? but i can't find how to do this.

linghengqian · 2022-04-11T09:17:48Z

@wlklnn

The answer maybe is yes. Anyway, it's just changing the date. If you don't use DistSQL, you can call a static method in actualdatanodes.

actual-data-nodes: ds0.$->{com.lingh.LocalShardingDatabasesAndTablesUtil.getActualDataNodes()}

 public class LocalShardingDatabasesAndTablesUtil { 
     public static List<String> getActualDataNodes() { 
         LocalDate startTime = LocalDate.now().minusDays(10); 
         LocalDate endTime = LocalDate.now(); 
         return LongStream.range(0, ChronoUnit.DAYS.between(startTime, endTime)) 
                 .mapToObj(startTime::plusDays) 
                 .map(localDate -> "table_name_" + localDate.format(DateTimeFormatter.ofPattern("yyyyMMdd"))) 
                 .collect(Collectors.toList()); 
     }
}

The simple choice is to set it directly in ShardingSphere JDBC, and then configure a cron pipeline task in gitlab to re-execute .gitlab-ci.yml every day, which requires a stage in the build and deploy projects (well, it seems very Boring, after all, everyone has been exposed to CI/CD).
Therefore, a good practice without restarting the project is to use ShardingSphere's DistSQL, and splicing the strings of actualdatanodes does not belong to the scope of how to do.
The reason why I still use JDK8's API demo is because it's very intuitive. I prefer to use java.time.LocalDate#datesUntil after JDK9 to simplify this long Stream() operation, but that's not what I mainly want to express.
I just wanted to express that there is absolutely no one-size-fits-all answer, but there are many kinds of answers, no?🕺🤳

wlklnn · 2022-04-11T09:36:21Z

@linghengqian
Thanks very much.
I just use ShardingSphere's jdbc ,not proxy ,so there is no DistSQL, and the most hard for me is :how to refresh actualdatanodes without restart my application. how to get the [actualdatanodes ] bean object and reset it's value is still unknown to me.
thanks agin for your reply.

linghengqian · 2022-04-11T09:52:16Z

There is also some experiment in ShardingSphere 4.x to implement SPI in the registry to dynamically modify actualDatanodes, this process becomes more complex in 5.x as the data distribution changes and the ShardingRule is rebuilt, I wonder if anyone's willing to do that. If you dispense with ShardingSphere data sources altogether, you can use its algorithm classes to encapsulate util yourself, bypassing the problems actualDatanodes needs to find.

wlklnn · 2022-04-13T09:52:06Z

I try to refresh it by ContextManager.alterRuleConfiguration(final String schemaName, final Collection ruleConfigs),It seems to be working properly.
InitActualDataNodesExample.txt

linghengqian · 2022-04-13T12:27:49Z

@wlklnn

Your sharing looks very cool! It's the first time I see such an application.

@strongduanmu

Is it possible to provide a Util class within ShardingSphere JDBC that wraps the method updateShardRuleActualDataNodes() for updating actual-data-nodes mentioned by @wlklnn ? I believe many people are looking for how to update it dynamically, and it is convenient to simplify the process.

package com.lingh.ao;

import org.apache.shardingsphere.driver.jdbc.core.datasource.ShardingSphereDataSource;
import org.apache.shardingsphere.infra.config.RuleConfiguration;
import org.apache.shardingsphere.sharding.algorithm.config.AlgorithmProvidedShardingRuleConfiguration;
import org.apache.shardingsphere.sharding.api.config.rule.ShardingTableRuleConfiguration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;
import java.util.Collection;
import java.util.LinkedList;

@Component
public class InitActualDataNodesAO {
    @Resource
    private ShardingSphereDataSource shardingSphereDataSource;
    private final String logicTableNameForTOrder = "t_order";
    private final String schemaNameForTOrder = "sharding_db";

    public void testSharding() {
        // generate actualDataNodes
        String actualDataNodes = "ds-0.t_order_$->{[1..8]}";

        this.updateShardRuleActualDataNodes(shardingSphereDataSource, schemaNameForTOrder, logicTableNameForTOrder, actualDataNodes);
    }

    private void updateShardRuleActualDataNodes(ShardingSphereDataSource dataSource, String schemaName, String logicTableName, String newActualDataNodes) {
        // Context manager.
        org.apache.shardingsphere.mode.manager.ContextManager contextManager = dataSource.getContextManager();

        // Rule configuration.
        Collection<RuleConfiguration> newRuleConfigList = new LinkedList<>();
        Collection<RuleConfiguration> oldRuleConfigList = dataSource.getContextManager()
                .getMetaDataContexts()
                .getMetaData(schemaName)
                .getRuleMetaData()
                .getConfigurations();

        for (RuleConfiguration oldRuleConfig : oldRuleConfigList) {
            if (oldRuleConfig instanceof AlgorithmProvidedShardingRuleConfiguration) {

                // Algorithm provided sharding rule configuration
                AlgorithmProvidedShardingRuleConfiguration oldAlgorithmConfig = (AlgorithmProvidedShardingRuleConfiguration) oldRuleConfig;
                AlgorithmProvidedShardingRuleConfiguration newAlgorithmConfig = new AlgorithmProvidedShardingRuleConfiguration();

                // Sharding table rule configuration Collection
                Collection<ShardingTableRuleConfiguration> newTableRuleConfigList = new LinkedList<>();
                Collection<ShardingTableRuleConfiguration> oldTableRuleConfigList = oldAlgorithmConfig.getTables();

                oldTableRuleConfigList.forEach(oldTableRuleConfig -> {
                    if (logicTableName.equals(oldTableRuleConfig.getLogicTable())) {
                        ShardingTableRuleConfiguration newTableRuleConfig = new ShardingTableRuleConfiguration(oldTableRuleConfig.getLogicTable(), newActualDataNodes);
                        newTableRuleConfig.setTableShardingStrategy(oldTableRuleConfig.getTableShardingStrategy());
                        newTableRuleConfig.setDatabaseShardingStrategy(oldTableRuleConfig.getDatabaseShardingStrategy());
                        newTableRuleConfig.setKeyGenerateStrategy(oldTableRuleConfig.getKeyGenerateStrategy());

                        newTableRuleConfigList.add(newTableRuleConfig);
                    } else {
                        newTableRuleConfigList.add(oldTableRuleConfig);
                    }
                });

                newAlgorithmConfig.setTables(newTableRuleConfigList);
                newAlgorithmConfig.setAutoTables(oldAlgorithmConfig.getAutoTables());
                newAlgorithmConfig.setBindingTableGroups(oldAlgorithmConfig.getBindingTableGroups());
                newAlgorithmConfig.setBroadcastTables(oldAlgorithmConfig.getBroadcastTables());
                newAlgorithmConfig.setDefaultDatabaseShardingStrategy(oldAlgorithmConfig.getDefaultDatabaseShardingStrategy());
                newAlgorithmConfig.setDefaultTableShardingStrategy(oldAlgorithmConfig.getDefaultTableShardingStrategy());
                newAlgorithmConfig.setDefaultKeyGenerateStrategy(oldAlgorithmConfig.getDefaultKeyGenerateStrategy());
                newAlgorithmConfig.setDefaultShardingColumn(oldAlgorithmConfig.getDefaultShardingColumn());
                newAlgorithmConfig.setShardingAlgorithms(oldAlgorithmConfig.getShardingAlgorithms());
                newAlgorithmConfig.setKeyGenerators(oldAlgorithmConfig.getKeyGenerators());

                newRuleConfigList.add(newAlgorithmConfig);
            }
        }

        // update context
        contextManager.alterRuleConfiguration(schemaName, newRuleConfigList);
    }
}

strongduanmu · 2022-04-13T13:53:25Z

Is it possible to provide a Util class within ShardingSphere JDBC that wraps the method updateShardRuleActualDataNodes() for updating actual-data-nodes mentioned by @wlklnn ? I believe many people are looking for how to update it dynamically, and it is convenient to simplify the process.

@linghengqian Thank you for your remind, I will take a look at this way.

xieyongchao2 · 2022-06-13T08:58:25Z

@strongduanmu @linghengqian
I found that I used version 5.1.1. The actual nodes are {[20220612,20220613]}. Only 20220612 nodes are configured through sharding JDBC. However, when I query, the node I landed on is 20220613 through the fragmentation algorithm, but there seems to be no error, and the data can be found.

linghengqian · 2022-06-13T09:04:22Z

@xieyongchao2 Replying to the closed issue is an inexplicable thing for the owner of this issue. I think it would be better if you open a new issue and provide a minimal replica demo, and then point out the association with this issue.

caol64 · 2022-08-12T01:53:46Z

@wlklnn

Your sharing looks very cool! It's the first time I see such an application.

@strongduanmu

Is it possible to provide a Util class within ShardingSphere JDBC that wraps the method updateShardRuleActualDataNodes() for updating actual-data-nodes mentioned by @wlklnn ? I believe many people are looking for how to update it dynamically, and it is convenient to simplify the process.

package com.lingh.ao;

import org.apache.shardingsphere.driver.jdbc.core.datasource.ShardingSphereDataSource;
import org.apache.shardingsphere.infra.config.RuleConfiguration;
import org.apache.shardingsphere.sharding.algorithm.config.AlgorithmProvidedShardingRuleConfiguration;
import org.apache.shardingsphere.sharding.api.config.rule.ShardingTableRuleConfiguration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;
import java.util.Collection;
import java.util.LinkedList;

@Component
public class InitActualDataNodesAO {
    @Resource
    private ShardingSphereDataSource shardingSphereDataSource;
    private final String logicTableNameForTOrder = "t_order";
    private final String schemaNameForTOrder = "sharding_db";

    public void testSharding() {
        // generate actualDataNodes
        String actualDataNodes = "ds-0.t_order_$->{[1..8]}";

        this.updateShardRuleActualDataNodes(shardingSphereDataSource, schemaNameForTOrder, logicTableNameForTOrder, actualDataNodes);
    }

    private void updateShardRuleActualDataNodes(ShardingSphereDataSource dataSource, String schemaName, String logicTableName, String newActualDataNodes) {
        // Context manager.
        org.apache.shardingsphere.mode.manager.ContextManager contextManager = dataSource.getContextManager();

        // Rule configuration.
        Collection<RuleConfiguration> newRuleConfigList = new LinkedList<>();
        Collection<RuleConfiguration> oldRuleConfigList = dataSource.getContextManager()
                .getMetaDataContexts()
                .getMetaData(schemaName)
                .getRuleMetaData()
                .getConfigurations();

        for (RuleConfiguration oldRuleConfig : oldRuleConfigList) {
            if (oldRuleConfig instanceof AlgorithmProvidedShardingRuleConfiguration) {

                // Algorithm provided sharding rule configuration
                AlgorithmProvidedShardingRuleConfiguration oldAlgorithmConfig = (AlgorithmProvidedShardingRuleConfiguration) oldRuleConfig;
                AlgorithmProvidedShardingRuleConfiguration newAlgorithmConfig = new AlgorithmProvidedShardingRuleConfiguration();

                // Sharding table rule configuration Collection
                Collection<ShardingTableRuleConfiguration> newTableRuleConfigList = new LinkedList<>();
                Collection<ShardingTableRuleConfiguration> oldTableRuleConfigList = oldAlgorithmConfig.getTables();

                oldTableRuleConfigList.forEach(oldTableRuleConfig -> {
                    if (logicTableName.equals(oldTableRuleConfig.getLogicTable())) {
                        ShardingTableRuleConfiguration newTableRuleConfig = new ShardingTableRuleConfiguration(oldTableRuleConfig.getLogicTable(), newActualDataNodes);
                        newTableRuleConfig.setTableShardingStrategy(oldTableRuleConfig.getTableShardingStrategy());
                        newTableRuleConfig.setDatabaseShardingStrategy(oldTableRuleConfig.getDatabaseShardingStrategy());
                        newTableRuleConfig.setKeyGenerateStrategy(oldTableRuleConfig.getKeyGenerateStrategy());

                        newTableRuleConfigList.add(newTableRuleConfig);
                    } else {
                        newTableRuleConfigList.add(oldTableRuleConfig);
                    }
                });

                newAlgorithmConfig.setTables(newTableRuleConfigList);
                newAlgorithmConfig.setAutoTables(oldAlgorithmConfig.getAutoTables());
                newAlgorithmConfig.setBindingTableGroups(oldAlgorithmConfig.getBindingTableGroups());
                newAlgorithmConfig.setBroadcastTables(oldAlgorithmConfig.getBroadcastTables());
                newAlgorithmConfig.setDefaultDatabaseShardingStrategy(oldAlgorithmConfig.getDefaultDatabaseShardingStrategy());
                newAlgorithmConfig.setDefaultTableShardingStrategy(oldAlgorithmConfig.getDefaultTableShardingStrategy());
                newAlgorithmConfig.setDefaultKeyGenerateStrategy(oldAlgorithmConfig.getDefaultKeyGenerateStrategy());
                newAlgorithmConfig.setDefaultShardingColumn(oldAlgorithmConfig.getDefaultShardingColumn());
                newAlgorithmConfig.setShardingAlgorithms(oldAlgorithmConfig.getShardingAlgorithms());
                newAlgorithmConfig.setKeyGenerators(oldAlgorithmConfig.getKeyGenerators());

                newRuleConfigList.add(newAlgorithmConfig);
            }
        }

        // update context
        contextManager.alterRuleConfiguration(schemaName, newRuleConfigList);
    }
}

@linghengqian In 5.1.2 version, ShardingSphereDataSource doesn't have getContextManager method. Is this solution needs to update?

linghengqian · 2022-08-12T03:22:22Z

@caol64

Hi, if you feel the need, you should open a new issue instead of staying in this one.
According to Confusion about changes to ShardingSphereDataSource and ContextManager #18494, the metadata module is undergoing a massive adjustment, which means that every version uses this method differently. I don't really recommend doing this, DistSQL is currently my solution.
If you wish to do this, you'll need to track the git changelog after each version update, like the issue I pointed out. Unless you are a source code reading enthusiast, this is actually very troublesome. Major changes to metadata in the matser branch will facilitate resolution at Exploration of ShardingSphere's base support for R2DBC SPI 1.0.0.RELEASE #19703.

caol64 · 2022-08-12T03:45:22Z

@linghengqian Understood, thanks.

SDSmalin · 2023-05-22T05:56:10Z

I found a simpler code

actual-data-nodes: ds0.$->{com.lingh.LocalShardingDatabasesAndTablesUtil.getActualDataNodes()}

public class LocalShardingDatabasesAndTablesUtil { 
     public static List<String> getActualDataNodes() { 
         LocalDate startTime = LocalDate.now().minusDays(10); 
         LocalDate endTime = LocalDate.now(); 
         return LongStream.range(0, ChronoUnit.DAYS.between(startTime, endTime)) 
                 .mapToObj(startTime::plusDays) 
                 .map(localDate -> "table_name_" + localDate.format(DateTimeFormatter.ofPattern("yyyyMMdd"))) 
                 .collect(Collectors.toList()); 
     }
}

public class InitActualDataNodesAO {
    @Resource
    private ShardingSphereDataSource shardingSphereDataSource;
    private final String schemaNameForTOrder = "sharding_db";

    public void testSharding() {
        this.reloadShardRuleActualDataNodes(shardingSphereDataSource, schemaNameForTOrder);
    }

    private void reloadShardRuleActualDataNodes(ShardingSphereDataSource dataSource, String schemaName) {
        // Context manager.
        org.apache.shardingsphere.mode.manager.ContextManager contextManager = dataSource.getContextManager();

        // Rule configuration.
        Collection<RuleConfiguration> ruleConfigList = dataSource.getContextManager()
                .getMetaDataContexts()
                .getMetaData(schemaName)
                .getRuleMetaData()
                .getConfigurations();
        // update context
        contextManager.alterRuleConfiguration(schemaName, ruleConfigList);
    }
}

It's just need to execute contextManager.alterRuleConfiguration() to reload actual-data-nodes

StarHuzy · 2023-10-09T03:41:15Z

There is also some experiment in ShardingSphere 4.x to implement SPI in the registry to dynamically modify actualDatanodes, this process becomes more complex in 5.x as the data distribution changes and the ShardingRule is rebuilt, I wonder if anyone's willing to do that. If you dispense with ShardingSphere data sources altogether, you can use its algorithm classes to encapsulate util yourself, bypassing the problems actualDatanodes needs to find.

hello , i Use custom table splitting strategy ，Successfully dynamically adding nodes
There will be no issues in the SQL execution of a single tables，But if there are node configuration issues in cascading queries with join query

@Component
@Slf4j
public class DataShardingAlgorithm implements StandardShardingAlgorithm<Long> {


    @Override
    public Collection<String> doSharding(Collection<String> collection, RangeShardingValue<Long> rangeShardingValue) {
        System.out.println("demo = " + collection);
        return collection;
    }

    @Override
    public String doSharding(Collection<String> collection, PreciseShardingValue<Long> preciseShardingValue) {
        StringBuilder resultTableName = new StringBuilder();
        String logicTableName = preciseShardingValue.getLogicTableName();
        // 拼接的tenantId，格式为 表名_{tenant_id}
        resultTableName.append(logicTableName).append("_").append(preciseShardingValue.getValue());
        String newTableName = resultTableName.toString().toLowerCase();
        if (!collection.contains(newTableName)) {
            // 动态新增节点
            ShardingAlgorithmTool.copyTable(logicTableName,newTableName);
            collection.add(newTableName);
        }
        System.out.println("collection = " + collection);
        return newTableName;
    }
    @Override
    public String getType() {
        return null;
    }
    @Override
    public void init(Properties properties) {

    }

}

linghengqian · 2023-10-09T11:53:33Z

@StarHuzy Hi, since this issue has long been closed, the content discussed above is no longer of any currency as a large number of API changes have been made. Please open a new issue for discussion.

StarHuzy · 2023-10-10T02:58:26Z

@StarHuzy Hi, since this issue has long been closed, the content discussed above is no longer of any currency as a large number of API changes have been made. Please open a new issue for discussion.
@linghengqian
Sorry for the slow response，I tried to explain him in simple and detailed terms.
This is my problem dynamic actual-data-nodes refresh in join query is invalid #28704 It was written in great detail and included the code ，Thank you very much for taking the time to answer， I think this is a significant discovery

strongduanmu added the type: question label Apr 11, 2022

wlklnn closed this as completed Apr 13, 2022

jianzh5 mentioned this issue Nov 13, 2022

KeyGenerator COSID_SNOWFLAKE does not take effect #22140

Closed

RaigorJiang mentioned this issue Jan 29, 2023

There was a problem upgrading the version 4.1.1 ~5.x #17146

Closed

liuzg459 mentioned this issue Jan 30, 2023

When the Manager. alterRuleConfiguration (dataSourceName, configurations) method is used, the new schema is empty #23839

Closed

zigaai mentioned this issue Mar 12, 2023

how can i refresh the DataNodes when i add a new dataSources #24562

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

how to config actual-data-nodes while my actual-data-nodes tables is dynamic。 #16725

how to config actual-data-nodes while my actual-data-nodes tables is dynamic。 #16725

wlklnn commented Apr 11, 2022

wlklnn commented Apr 11, 2022

linghengqian commented Apr 11, 2022

wlklnn commented Apr 11, 2022

linghengqian commented Apr 11, 2022 •

edited

Loading

wlklnn commented Apr 11, 2022

linghengqian commented Apr 11, 2022

wlklnn commented Apr 13, 2022

linghengqian commented Apr 13, 2022

strongduanmu commented Apr 13, 2022

xieyongchao2 commented Jun 13, 2022

linghengqian commented Jun 13, 2022

caol64 commented Aug 12, 2022

linghengqian commented Aug 12, 2022 •

edited

Loading

caol64 commented Aug 12, 2022

SDSmalin commented May 22, 2023

StarHuzy commented Oct 9, 2023

linghengqian commented Oct 9, 2023

StarHuzy commented Oct 10, 2023

how to config actual-data-nodes while my actual-data-nodes tables is dynamic。 #16725

how to config actual-data-nodes while my actual-data-nodes tables is dynamic。 #16725

Comments

wlklnn commented Apr 11, 2022

wlklnn commented Apr 11, 2022

linghengqian commented Apr 11, 2022

wlklnn commented Apr 11, 2022

linghengqian commented Apr 11, 2022 • edited Loading

wlklnn commented Apr 11, 2022

linghengqian commented Apr 11, 2022

wlklnn commented Apr 13, 2022

linghengqian commented Apr 13, 2022

strongduanmu commented Apr 13, 2022

xieyongchao2 commented Jun 13, 2022

linghengqian commented Jun 13, 2022

caol64 commented Aug 12, 2022

linghengqian commented Aug 12, 2022 • edited Loading

caol64 commented Aug 12, 2022

SDSmalin commented May 22, 2023

StarHuzy commented Oct 9, 2023

linghengqian commented Oct 9, 2023

StarHuzy commented Oct 10, 2023

linghengqian commented Apr 11, 2022 •

edited

Loading

linghengqian commented Aug 12, 2022 •

edited

Loading