I am running a Kubernetes cluster in AWS EC2 instances. I have one master node (t2.2large EC2 Instance) and two worker nodes (t2.xlarge EC2 instance). I created the cluster using KubeADM. The cluster is running fine.
I need to setup OpenSearch Cluster and Dashboard in my Kubernetes cluster. I was following the official documentation: https://opensearch.org/docs/latest/install-and-configure/install-opensearch/helm/
Once the OpenSearch cluster pods get created, they are in pending state and are not getting into running state.
When I described one of the pods, I am getting the below event logs:
Events: Type Reason Age From Message
Warning FailedScheduling 41s default-scheduler 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling..
When I checked the PersistentVolumeClaim, I am getting the below three PVCs and seems that they are in pending state.
When I describe the PVC, I am getting the below event log:
Normal FailedBinding 0s (x7 over 88s) persistentvolume-controller no persistent volumes available for this claim and no storage class is set.
And when I checked for any PersistentVolume, I am not seeing any. The OpenSearch cluster was created using the HELM chart from the official documentation, and it seems that the associated PVs are not getting created. Then I manually created three PVs and attached them to the PVCs and the above error got resolved.
However, even after that I was facing some issues. It seems that the pods are in CrashLoopBackOff or Error state and they are in 0/1 state.
When I described one of the pods, I am getting the below error:
Events: Type Reason Age From Message
Warning Unhealthy 6m29s (x12 over 58m) kubelet Startup probe failed: dial tcp 10.1.0.18:9200: connect: connection refused Warning BackOff 3m19s (x111 over 57m) kubelet Back-off restarting failed container opensearch in pod opensearch-cluster-master-0_default(d7d9fdc6-88ec-47e5-9420-c9a8dacf9ae1)
Seems that the the pod is trying to communicate to a service serving port 9200 and there's a connection refused issue.
After checking the service, I am getting the below result:
Seems that the OpenSearch master-nodes(Actually pods) are trying to connect to 10.1.0.18:9200, However, from the screen shot that I have attached, the Service ClusterIP of opensearch-cluster-master is 10.99.173.27 and it's serving port 9200. That's one area where I am confused. Also, seems that there's no option in the values.yaml or CRD to specify the value of the clusterIP.
Below is the logs of one of the OpenSearch Master pod:
Defaulted container "opensearch" out of: opensearch, fsgroup-volume (init)
Enabling execution of install_demo_configuration.sh for OpenSearch Security Plugin
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **
**************************************************************************
OpenSearch Security Demo Installer
** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
OpenSearch install type: rpm/deb on Amazon Linux release 2023 (Amazon Linux)
OpenSearch config dir: /usr/share/opensearch/config
OpenSearch config file: /usr/share/opensearch/config/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.11.0
Detected OpenSearch Security Version: 2.11.0.0
tee: /usr/share/opensearch/config/opensearch.yml: Permission denied
Enabling OpenSearch Security Plugin
Enabling execution of OPENSEARCH_HOME/bin/opensearch-performance-analyzer/performance-analyzer-agent-cli for OpenSearch Performance Analyzer Plugin
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch/lib/opensearch-2.11.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
WARNING: System::setSecurityManager will be removed in a future release
WARNING: A terminally deprecated method in java.lang.System has been called
WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/opensearch/lib/opensearch-2.11.0.jar)
WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
WARNING: System::setSecurityManager will be removed in a future release
[2023-11-23T10:05:41,318][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] version[2.11.0], pid[48], build[tar/4dcad6dd1fd45b6bd91f041a041829c8687278fa/ 2023-10-13T02:55:55.511945994Z], OS[Linux/6.4.16-linuxkit/amd64], JVM[Eclipse Adoptium/OpenJDK 64-Bit Server VM/17.0.8/17.0.8+7]
[2023-11-23T10:05:41,324][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2023-11-23T10:05:41,324][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch. networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX: +ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator. numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5367794133992170925, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/ hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org. opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/ opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xmx512M, -Xms512M, -XX:MaxDirectMemorySize=268435456, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution. type=tar, -Dopensearch.bundled_jdk=true]
[2023-11-23T10:05:43,188][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2023-11-23T10:05:43,189][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] OpenSearch Config path is /usr/share/opensearch/config
[2023-11-23T10:05:43,568][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] JVM supports TLSv1.3
[2023-11-23T10:05:43,570][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Config directory is /usr/share/opensearch/config/, from there the key- and truststore files are resolved relatively
[2023-11-23T10:05:44,187][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS Transport Client Provider : JDK
[2023-11-23T10:05:44,188][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS Transport Server Provider : JDK
[2023-11-23T10:05:44,188][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] TLS HTTP Provider : JDK
[2023-11-23T10:05:44,188][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-11-23T10:05:44,188][INFO ][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2023-11-23T10:05:44,251][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Clustername: opensearch-cluster
[2023-11-23T10:05:44,261][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
[2023-11-23T10:05:44,262][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
[2023-11-23T10:05:45,138][INFO ][o.o.p.c.c.PluginSettings ] [opensearch-cluster-master-0] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-11-23T10:05:45,826][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-cluster-master-0] ReindexPlugin reloadSPI called
[2023-11-23T10:05:45,827][INFO ][o.o.i.r.ReindexPlugin ] [opensearch-cluster-master-0] Unable to find any implementation for RemoteReindexExtension
[2023-11-23T10:05:45,911][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-11-23T10:05:45,917][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro_anomaly_detector, index: . opendistro-anomaly-detector-jobs
[2023-11-23T10:05:45,918][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: opendistro-index-management, index: . opendistro-ism-config
[2023-11-23T10:05:45,919][INFO ][o.o.j.JobSchedulerPlugin ] [opensearch-cluster-master-0] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: . scheduler-geospatial-ip2geo-datasource
[2023-11-23T10:05:45,928][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [aggs-matrix-stats]
[2023-11-23T10:05:45,929][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [analysis-common]
[2023-11-23T10:05:45,929][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [geo]
[2023-11-23T10:05:45,929][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-common]
[2023-11-23T10:05:45,929][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-geoip]
[2023-11-23T10:05:45,930][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [ingest-user-agent]
[2023-11-23T10:05:45,930][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-expression]
[2023-11-23T10:05:45,930][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-mustache]
[2023-11-23T10:05:45,930][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [lang-painless]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [mapper-extras]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [opensearch-dashboards]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [parent-join]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [percolator]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [rank-eval]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [reindex]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [repository-url]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [search-pipeline-common]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [systemd]
[2023-11-23T10:05:45,933][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded module [transport-netty4]
[2023-11-23T10:05:45,934][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-alerting]
[2023-11-23T10:05:45,934][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-anomaly-detection]
[2023-11-23T10:05:45,934][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-asynchronous-search]
[2023-11-23T10:05:45,934][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-cross-cluster-replication]
[2023-11-23T10:05:45,935][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-custom-codecs]
[2023-11-23T10:05:45,935][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-geospatial]
[2023-11-23T10:05:45,935][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-index-management]
[2023-11-23T10:05:45,935][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-job-scheduler]
[2023-11-23T10:05:45,935][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-knn]
[2023-11-23T10:05:45,936][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-ml]
[2023-11-23T10:05:45,936][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-neural-search]
[2023-11-23T10:05:45,936][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-notifications]
[2023-11-23T10:05:45,936][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-notifications-core]
[2023-11-23T10:05:45,936][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-observability]
[2023-11-23T10:05:45,936][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-performance-analyzer]
[2023-11-23T10:05:45,936][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-reports-scheduler]
[2023-11-23T10:05:45,937][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-security]
[2023-11-23T10:05:45,937][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-security-analytics]
[2023-11-23T10:05:45,937][INFO ][o.o.p.PluginsService ] [opensearch-cluster-master-0] loaded plugin [opensearch-sql]
[2023-11-23T10:05:45,999][INFO ][o.o.s.OpenSearchSecurityPlugin] [opensearch-cluster-master-0] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2023-11-23T10:05:46,003][INFO ][o.o.e.ExtensionsManager ] [opensearch-cluster-master-0] ExtensionsManager initialized
[2023-11-23T10:05:46,025][INFO ][o.o.e.NodeEnvironment ] [opensearch-cluster-master-0] using [1] data paths, mounts [[/usr/share/opensearch/data (/dev/vda1)]], net usable_space [51.1gb], net total_space [58.3gb], types [ext4]
[2023-11-23T10:05:46,026][INFO ][o.o.e.NodeEnvironment ] [opensearch-cluster-master-0] heap size [512mb], compressed ordinary object pointers [true]
[2023-11-23T10:05:46,034][DEPRECATION][o.o.d.c.n.DiscoveryNodeRole] [opensearch-cluster-master-0] Assigning [master] role in setting [node.roles] is deprecated. To promote inclusive language, please use [cluster_manager] role instead.
[2023-11-23T10:05:46,131][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] node name [opensearch-cluster-master-0], node ID [FaICBDbARHSCjAtbEHwNug], cluster name [opensearch-cluster], roles [ingest, remote_cluster_client, master, data]
[2023-11-23T10:05:49,157][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-cluster-master-0] [index.store.hybrid.mmap.extensions] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-11-23T10:05:49,159][DEPRECATION][o.o.d.c.s.Settings ] [opensearch-cluster-master-0] [cluster.initial_master_nodes] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2023-11-23T10:05:50,306][INFO ][o.o.n.p.NeuralSearch ] [opensearch-cluster-master-0] Registering hybrid query phase searcher with feature flag [plugins.neural_search. hybrid_search_disabled]
[2023-11-23T10:05:50,652][WARN ][o.o.s.c.Salt ] [opensearch-cluster-master-0] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2023-11-23T10:05:50,703][INFO ][o.o.s.a.i.AuditLogImpl ] [opensearch-cluster-master-0] Message routing enabled: true
[2023-11-23T10:05:50,751][INFO ][o.o.s.f.SecurityFilter ] [opensearch-cluster-master-0] <NONE> indices are made immutable.
[2023-11-23T10:05:51,283][INFO ][o.o.a.b.ADCircuitBreakerService] [opensearch-cluster-master-0] Registered memory breaker.
[2023-11-23T10:05:51,850][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML memory breaker.
[2023-11-23T10:05:51,851][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML disk breaker.
[2023-11-23T10:05:51,851][INFO ][o.o.m.b.MLCircuitBreakerService] [opensearch-cluster-master-0] Registered ML native memory breaker.
[2023-11-23T10:05:51,962][INFO ][o.r.Reflections ] [opensearch-cluster-master-0] Reflections took 65 ms to scan 1 urls, producing 20 keys and 51 values
[2023-11-23T10:05:52,075][WARN ][o.o.s.p.SQLPlugin ] [opensearch-cluster-master-0] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/ sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2023-11-23T10:05:52,116][WARN ][o.o.s.p.SQLPlugin ] [opensearch-cluster-master-0] Async Query APIs are disabled as plugins.query.executionengine.spark.config is not configured properly in cluster settings. Please configure and restart the domain to enable Async Query APIs
[2023-11-23T10:05:52,767][INFO ][o.o.t.NettyAllocator ] [opensearch-cluster-master-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=512mb}]
[2023-11-23T10:05:52,771][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled
[2023-11-23T10:05:52,894][INFO ][o.o.d.DiscoveryModule ] [opensearch-cluster-master-0] using discovery type [zen] and seed hosts providers [settings]
[2023-11-23T10:05:53,467][WARN ][o.o.g.DanglingIndicesState] [opensearch-cluster-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2023-11-23T10:05:53,990][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [opensearch-cluster-master-0] PerformanceAnalyzer Enabled: false
[2023-11-23T10:05:54,023][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] initialized
[2023-11-23T10:05:54,023][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] starting ...
[2023-11-23T10:05:54,052][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [windows_logtype.json] log type
[2023-11-23T10:05:54,053][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [waf_logtype.json] log type
[2023-11-23T10:05:54,053][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [vpcflow_logtype.json] log type
[2023-11-23T10:05:54,054][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [test_windows_logtype.json] log type
[2023-11-23T10:05:54,055][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [s3_logtype.json] log type
[2023-11-23T10:05:54,055][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_web_logtype.json] log type
[2023-11-23T10:05:54,056][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_proxy_logtype.json] log type
[2023-11-23T10:05:54,057][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_macos_logtype.json] log type
[2023-11-23T10:05:54,057][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_compliance_logtype.json] log type
[2023-11-23T10:05:54,057][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_cloud_logtype.json] log type
[2023-11-23T10:05:54,058][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_apt_logtype.json] log type
[2023-11-23T10:05:54,058][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [others_application_logtype.json] log type
[2023-11-23T10:05:54,059][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [okta_logtype.json] log type
[2023-11-23T10:05:54,060][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [network_logtype.json] log type
[2023-11-23T10:05:54,060][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [netflow_logtype.json] log type
[2023-11-23T10:05:54,060][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [m365_logtype.json] log type
[2023-11-23T10:05:54,061][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [linux_logtype.json] log type
[2023-11-23T10:05:54,061][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [gworkspace_logtype.json] log type
[2023-11-23T10:05:54,061][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [github_logtype.json] log type
[2023-11-23T10:05:54,062][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [dns_logtype.json] log type
[2023-11-23T10:05:54,062][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [cloudtrail_logtype.json] log type
[2023-11-23T10:05:54,063][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [azure_logtype.json] log type
[2023-11-23T10:05:54,063][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [apache_access_logtype.json] log type
[2023-11-23T10:05:54,064][INFO ][o.o.s.l.BuiltinLogTypeLoader] [opensearch-cluster-master-0] Loaded [ad_ldap_logtype.json] log type
[2023-11-23T10:05:54,166][INFO ][o.o.t.TransportService ] [opensearch-cluster-master-0] publish_address {10.1.0.18:9300}, bound_addresses {0.0.0.0:9300}
[2023-11-23T10:05:54,169][INFO ][o.o.t.TransportService ] [opensearch-cluster-master-0] Remote clusters initialized successfully.
[2023-11-23T10:05:54,308][INFO ][o.o.b.BootstrapChecks ] [opensearch-cluster-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
Below is the configuration for the PVs:
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: opensearch-cluster-master-opensearch-cluster-master-0-pv-2
namespace: open-search
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 30Gi
accessModes:
- ReadWriteOnce
claimRef:
namespace: open-search
name: opensearch-cluster-master-opensearch-cluster-master-0
hostPath:
path: "/data"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: opensearch-cluster-master-opensearch-cluster-master-1-pv-2
namespace: open-search
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 30Gi
accessModes:
- ReadWriteOnce
claimRef:
namespace: open-search
name: opensearch-cluster-master-opensearch-cluster-master-1
hostPath:
path: "/data"
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: opensearch-cluster-master-opensearch-cluster-master-2-pv-2
namespace: open-search
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 30Gi
accessModes:
- ReadWriteOnce
claimRef:
namespace: open-search
name: opensearch-cluster-master-opensearch-cluster-master-2
hostPath:
path: "/data"
Below is the configuration for the PVCs (after running kubectl get pvc -n open-search opensearch-cluster-master-opensearch-cluster-master-0 -o yaml) that got automatically created by HELM:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2023-11-23T13:46:31Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app.kubernetes.io/instance: opensearch-cluster
app.kubernetes.io/name: opensearch
name: opensearch-cluster-master-opensearch-cluster-master-0
namespace: open-search
resourceVersion: "2407225"
uid: ba8243d4-d1d7-4d42-bb06-597d678f149c
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
volumeMode: Filesystem
volumeName: opensearch-cluster-master-opensearch-cluster-master-0-pv
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 30Gi
phase: Bound
Does anyone know how to fix this? I tried with different types of Kubernetes clusters including Docker-For-Desktop, AKS, etc. Yet still I am getting the same error.
