Skip to content

HDDS-14518. Support open keys and MPU in ContainerToKeyMapping tool#9703

Open
sarvekshayr wants to merge 3 commits intoapache:masterfrom
sarvekshayr:HDDS-14518
Open

HDDS-14518. Support open keys and MPU in ContainerToKeyMapping tool#9703
sarvekshayr wants to merge 3 commits intoapache:masterfrom
sarvekshayr:HDDS-14518

Conversation

@sarvekshayr
Copy link
Contributor

What changes were proposed in this pull request?

Added a flag --in-progress to include open files/keys and multipart uploads in ContainerToKeyMapping tool.
When --onlyFileNames is set, then add FSO and OBS tags to differentiate the keys.

What is the link to the Apache JIRA

HDDS-14518

How was this patch tested?

Only shows files and keys from fileTable and keyTable respectively.

ozone debug om container-key-mapping --db /data/metadata/om.db --containers 1,2,3,4
{
  "containers" : {
    "1" : {
      "keys" : [ "/vol1/fso-bucket/dir1/file1" ],
      "totalKeys" : 1
    },
    "2" : {
      "keys" : [ "/vol1/obs-bucket/key1" ],
      "totalKeys" : 1
    },
    "3" : {
      "keys" : [ "[unreferenced] unreferencedFile" ],
      "totalKeys" : 1,
      "unreferencedKeys" : 1
    },
    "4" : {
      "keys" : [ ],
      "totalKeys" : 0
    }
  }
}

Shows open files and keys from openFileTable and openKeyTable respectively. Also includes multipart uploads info from multipartInfoTable

ozone debug om container-key-mapping --db /data/metadata/om.db --containers 1,2,3,4 --in-progress
{
  "containers" : {
    "1" : {
      "keys" : [ "/vol1/fso-bucket/dir1/file1" ],
      "openKeys" : [ "/vol1/fso-bucket/dir1/openFile" ],
      "totalKeys" : 2
    },
    "2" : {
      "keys" : [ "/vol1/obs-bucket/key1" ],
      "openKeys" : [ "/vol1/obs-bucket/openKey" ],
      "totalKeys" : 2
    },
    "3" : {
      "keys" : [ "[unreferenced] unreferencedFile" ],
      "openKeys" : [ ],
      "totalKeys" : 1,
      "unreferencedKeys" : 1
    },
    "4" : {
      "keys" : [ ],
      "openKeys" : [ "[mpu-part-1] /vol1/obs-bucket/mpuKey/test-upload-id/part-1", "[mpu-part-2] /vol1/obs-bucket/mpuKey/test-upload-id/part-2" ],
      "totalKeys" : 2
    }
  }
}

Show only file names

ozone debug om container-key-mapping --db /data/metadata/om.db --containers 1,2,3,4 --in-progress --onlyFileNames
{
  "containers" : {
    "1" : {
      "keys" : [ "[fso] file1" ],
      "openKeys" : [ "[fso] openFile" ],
      "totalKeys" : 2
    },
    "2" : {
      "keys" : [ "[obs] key1" ],
      "openKeys" : [ "[obs] openKey" ],
      "totalKeys" : 2
    },
    "3" : {
      "keys" : [ "[fso] unreferencedFile" ],
      "openKeys" : [ ],
      "totalKeys" : 1
    },
    "4" : {
      "keys" : [ ],
      "openKeys" : [ "[mpu-part-1-obs] mpuKey/test-upload-id/part-1", "[mpu-part-2-obs] mpuKey/test-upload-id/part-2" ],
      "totalKeys" : 2
    }
  }
}

Copy link
Contributor

@sreejasahithi sreejasahithi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sarvekshayr for working on this.
Could you please reduce the duplication between processOpenFiles() and processOpenKeys().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants