Skip to content

deploy 5 CockroachDB nodes from RSS #3450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jul 3, 2023
Merged

deploy 5 CockroachDB nodes from RSS #3450

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
5a7ecec
rework RSS sled allocation to be component-oriented
davepacheco Jun 28, 2023
9c7bbf7
Merge branch 'main' into dap/multi-crdb
davepacheco Jun 29, 2023
a3012b2
deploy 5 cockroachdb nodes
davepacheco Jun 29, 2023
2c30183
spread stuff across U.2 zpools too
davepacheco Jun 29, 2023
ee88853
do not need to unwrap so much
davepacheco Jun 29, 2023
409c949
create_virtual_hardware.sh will need to create two more zpools
davepacheco Jun 29, 2023
a3e0562
Merge remote-tracking branch 'origin/dap/multi-crdb' into dap/more-crdb
davepacheco Jun 29, 2023
5e5177a
dnswait should print port numbers too
davepacheco Jun 29, 2023
4d87e71
end-to-end tests will need to wait longer, emit more debug information
davepacheco Jun 29, 2023
64c0c4f
use smaller disks, add debug data
davepacheco Jun 30, 2023
8651d75
end-to-end test needs to wait longer (try 2)
davepacheco Jun 30, 2023
1976753
add some debugging
davepacheco Jun 30, 2023
d820a55
Merge branch 'dap/more-crdb' of github.com:oxidecomputer/omicron into…
davepacheco Jun 30, 2023
c24c2e8
flip order of Nexus/External DNS assignment
davepacheco Jun 30, 2023
752f01e
Merge remote-tracking branch 'origin/dap/multi-crdb' into dap/more-crdb
davepacheco Jun 30, 2023
d98a2a5
remove debugging stuff
davepacheco Jun 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions common/src/sql/dbinit.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@ CREATE DATABASE omicron;
CREATE USER omicron;
ALTER DEFAULT PRIVILEGES GRANT INSERT, SELECT, UPDATE, DELETE ON TABLES to omicron;

/*
* Configure a replication factor of 5 to ensure that the system can maintain
* availability in the face of any two node failures.
*/
ALTER RANGE default CONFIGURE ZONE USING num_replicas = 5;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For local testing CRDB -- e.g. with the omicron-dev tool -- what does this do?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question. I expect it will have no practical impact. The default replication factor is 3, which is already more nodes than we ever have today. The impact is that CockroachDB considers all of its data under-replicated. It reports it as such (e.g., here) and if new nodes show up, it will try to replicate data until nothing is under-replicated.

One could imagine that this would instead disallow you from writing any data to the cluster when it can't satisfy this constraint. That doesn't seem to be how it works. If it did, the tests wouldn't run at all, I think.


/*
* Racks
*/
Expand Down
4 changes: 2 additions & 2 deletions end-to-end-tests/src/helpers/ctx.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ pub async fn nexus_addr() -> Result<IpAddr> {
&resolver,
&dns_name,
Duration::from_secs(1),
Duration::from_secs(300),
Duration::from_secs(600),
)
.await
}
Expand Down Expand Up @@ -240,7 +240,7 @@ pub async fn build_client() -> Result<oxide_client::Client> {
})
},
&Duration::from_secs(1),
&Duration::from_secs(300),
&Duration::from_secs(600),
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're deploying two extra CockroachDB nodes and it can take a little longer for the system to set up now.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can look into parallelizing service bringup on the sled-agent side, if that would help mitigate?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I expect it would. No pressure on my part in terms of urgency.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, lets not block this PR, but I can do this as a follow-up.

)
.await
.context("logging in")?;
Expand Down
4 changes: 2 additions & 2 deletions internal-dns-cli/src/bin/dnswait.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,8 +90,8 @@ async fn main() -> Result<()> {
.await
.context("unexpectedly gave up")?;

for ip in result {
println!("{}", ip)
for (target, port) in result {
println!("{}:{}", target, port)
}

Ok(())
Expand Down
7 changes: 5 additions & 2 deletions internal-dns/src/resolver.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ impl Resolver {
pub async fn lookup_srv(
&self,
srv: crate::ServiceName,
) -> Result<Vec<String>, ResolveError> {
) -> Result<Vec<(String, u16)>, ResolveError> {
let name = format!("{}.{}", srv.dns_name(), DNS_ZONE);
trace!(self.log, "lookup_srv"; "dns_name" => &name);
let response = self.inner.srv_lookup(&name).await?;
Expand All @@ -148,7 +148,10 @@ impl Resolver {
"response" => ?response
);

Ok(response.into_iter().map(|srv| srv.target().to_string()).collect())
Ok(response
.into_iter()
.map(|srv| (srv.target().to_string(), srv.port()))
.collect())
}

pub async fn lookup_all_ipv6(
Expand Down
Loading