Fix graph building to exclude input, output and initializer from value_info

Fix graph building to exclude input, output and initializer from value_info #5631