[SRU,UNSTABLE/BIONIC/XENIAL] UBUNTU: SAUCE: overlayfs: use shiftfs hacks only with shiftfs as underlay

Submitted by Christian Brauner on April 23, 2020, 10:11 a.m.

Details

Message ID 20200423101128.298615-1-christian.brauner@ubuntu.com
State New
Series "UBUNTU: SAUCE: overlayfs: use shiftfs hacks only with shiftfs as underlay"
Headers show

Commit Message

Christian Brauner April 23, 2020, 10:11 a.m.
From: Andrei Vagin <avagin@gmail.com>

BugLink: https://bugs.launchpad.net/bugs/1857257

The hack was introduced in ("UBUNTU: SAUCE: overlayfs: allow with
shiftfs as underlay") and it broke checkpoint/restore of docker
contains:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1857257

The following script can be used to trigger the issue:
  #!/bin/bash

  cat > test.py << EOF
  import sys

  f = open("/proc/self/maps")

  for l in f.readlines():
    if "python" not in l:
      continue
    print(l)
    s = l.split()
    start, end = s[0].split("-")
    fname = s[-1]
    print(start, end, fname)
    break
  else:
    sys.exit(1)

  test_file1 = open(fname)
  test_file2 = open("/proc/self/map_files/%s-%s" % (start, end))

  fdinfo1 = open("/proc/self/fdinfo/%d" % test_file1.fileno()).read()
  fdinfo2 = open("/proc/self/fdinfo/%d" % test_file2.fileno()).read()

  if fdinfo1 != fdinfo2:
    print("FAIL")
    print(test_file1)
    print(fdinfo1)
    print(test_file2)
    print(fdinfo2)
    sys.exit(1)
  print("PASS")
  EOF
  sudo docker run -it --privileged --rm -v `pwd`:/mnt python python /mnt/test.py

Cc: Stefan Bader <stefan.bader@canonical.com>
Cc: Connor Kuehl <connor.kuehl@canonical.com>
Cc: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
Fixes: 58009298c6bd ("UBUNTU: SAUCE: overlayfs: allow with shiftfs as underlay")
Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
Signed-off-by: Andrei Vagin <avagin@gmail.com>
---
 fs/overlayfs/file.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)


base-commit: 48f168950cd193d9beedc0afb0c79a254e32f637

Patch hide | download patch | download mbox

diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
index 4f94ef663569..9d16fff5342a 100644
--- a/fs/overlayfs/file.c
+++ b/fs/overlayfs/file.c
@@ -42,9 +42,12 @@  static struct file *ovl_open_realfile(const struct file *file,
 	int flags = file->f_flags | O_NOATIME | FMODE_NONOTIFY;
 
 	old_cred = ovl_override_creds(inode->i_sb);
-	ovl_path_real(file->f_path.dentry, &realpath);
-	realfile = open_with_fake_path(&realpath, flags, realinode,
-				       current_cred());
+	if (realpath.dentry->d_sb->s_magic == SHIFTFS_MAGIC)
+		realfile = open_with_fake_path(&realpath, flags, realinode,
+					       current_cred());
+	else
+		realfile = open_with_fake_path(&file->f_path, flags, realinode,
+					       current_cred());
 	revert_creds(old_cred);
 
 	pr_debug("open(%p[%pD2/%c], 0%o) -> (%p, 0%o)\n",

Comments

Kleber Souza April 23, 2020, 10:31 a.m.
On 23.04.20 12:11, Christian Brauner wrote:
> From: Andrei Vagin <avagin@gmail.com>
> 
> BugLink: https://bugs.launchpad.net/bugs/1857257
> 
> The hack was introduced in ("UBUNTU: SAUCE: overlayfs: allow with
> shiftfs as underlay") and it broke checkpoint/restore of docker
> contains:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1857257
> 
> The following script can be used to trigger the issue:
>   #!/bin/bash
> 
>   cat > test.py << EOF
>   import sys
> 
>   f = open("/proc/self/maps")
> 
>   for l in f.readlines():
>     if "python" not in l:
>       continue
>     print(l)
>     s = l.split()
>     start, end = s[0].split("-")
>     fname = s[-1]
>     print(start, end, fname)
>     break
>   else:
>     sys.exit(1)
> 
>   test_file1 = open(fname)
>   test_file2 = open("/proc/self/map_files/%s-%s" % (start, end))
> 
>   fdinfo1 = open("/proc/self/fdinfo/%d" % test_file1.fileno()).read()
>   fdinfo2 = open("/proc/self/fdinfo/%d" % test_file2.fileno()).read()
> 
>   if fdinfo1 != fdinfo2:
>     print("FAIL")
>     print(test_file1)
>     print(fdinfo1)
>     print(test_file2)
>     print(fdinfo2)
>     sys.exit(1)
>   print("PASS")
>   EOF
>   sudo docker run -it --privileged --rm -v `pwd`:/mnt python python /mnt/test.py
> 
> Cc: Stefan Bader <stefan.bader@canonical.com>
> Cc: Connor Kuehl <connor.kuehl@canonical.com>
> Cc: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
> Fixes: 58009298c6bd ("UBUNTU: SAUCE: overlayfs: allow with shiftfs as underlay")
> Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
> Signed-off-by: Andrei Vagin <avagin@gmail.com>

Thanks Christian for re-sending it.

Acked-by: Kleber Sacilotto de Souza <kleber.souza@canonical.com>

> ---
>  fs/overlayfs/file.c | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> index 4f94ef663569..9d16fff5342a 100644
> --- a/fs/overlayfs/file.c
> +++ b/fs/overlayfs/file.c
> @@ -42,9 +42,12 @@ static struct file *ovl_open_realfile(const struct file *file,
>  	int flags = file->f_flags | O_NOATIME | FMODE_NONOTIFY;
>  
>  	old_cred = ovl_override_creds(inode->i_sb);
> -	ovl_path_real(file->f_path.dentry, &realpath);
> -	realfile = open_with_fake_path(&realpath, flags, realinode,
> -				       current_cred());
> +	if (realpath.dentry->d_sb->s_magic == SHIFTFS_MAGIC)
> +		realfile = open_with_fake_path(&realpath, flags, realinode,
> +					       current_cred());
> +	else
> +		realfile = open_with_fake_path(&file->f_path, flags, realinode,
> +					       current_cred());
>  	revert_creds(old_cred);
>  
>  	pr_debug("open(%p[%pD2/%c], 0%o) -> (%p, 0%o)\n",
> 
> base-commit: 48f168950cd193d9beedc0afb0c79a254e32f637
>
Kamal Mostafa April 23, 2020, 4:58 p.m.
Acked-by: Kamal Mostafa <kamal@canonical.com>

 -Kamal

On Thu, Apr 23, 2020 at 12:11:28PM +0200, Christian Brauner wrote:
> From: Andrei Vagin <avagin@gmail.com>
> 
> BugLink: https://bugs.launchpad.net/bugs/1857257
> 
> The hack was introduced in ("UBUNTU: SAUCE: overlayfs: allow with
> shiftfs as underlay") and it broke checkpoint/restore of docker
> contains:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1857257
> 
> The following script can be used to trigger the issue:
>   #!/bin/bash
> 
>   cat > test.py << EOF
>   import sys
> 
>   f = open("/proc/self/maps")
> 
>   for l in f.readlines():
>     if "python" not in l:
>       continue
>     print(l)
>     s = l.split()
>     start, end = s[0].split("-")
>     fname = s[-1]
>     print(start, end, fname)
>     break
>   else:
>     sys.exit(1)
> 
>   test_file1 = open(fname)
>   test_file2 = open("/proc/self/map_files/%s-%s" % (start, end))
> 
>   fdinfo1 = open("/proc/self/fdinfo/%d" % test_file1.fileno()).read()
>   fdinfo2 = open("/proc/self/fdinfo/%d" % test_file2.fileno()).read()
> 
>   if fdinfo1 != fdinfo2:
>     print("FAIL")
>     print(test_file1)
>     print(fdinfo1)
>     print(test_file2)
>     print(fdinfo2)
>     sys.exit(1)
>   print("PASS")
>   EOF
>   sudo docker run -it --privileged --rm -v `pwd`:/mnt python python /mnt/test.py
> 
> Cc: Stefan Bader <stefan.bader@canonical.com>
> Cc: Connor Kuehl <connor.kuehl@canonical.com>
> Cc: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
> Fixes: 58009298c6bd ("UBUNTU: SAUCE: overlayfs: allow with shiftfs as underlay")
> Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
> Signed-off-by: Andrei Vagin <avagin@gmail.com>
> ---
>  fs/overlayfs/file.c | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> index 4f94ef663569..9d16fff5342a 100644
> --- a/fs/overlayfs/file.c
> +++ b/fs/overlayfs/file.c
> @@ -42,9 +42,12 @@ static struct file *ovl_open_realfile(const struct file *file,
>  	int flags = file->f_flags | O_NOATIME | FMODE_NONOTIFY;
>  
>  	old_cred = ovl_override_creds(inode->i_sb);
> -	ovl_path_real(file->f_path.dentry, &realpath);
> -	realfile = open_with_fake_path(&realpath, flags, realinode,
> -				       current_cred());
> +	if (realpath.dentry->d_sb->s_magic == SHIFTFS_MAGIC)
> +		realfile = open_with_fake_path(&realpath, flags, realinode,
> +					       current_cred());
> +	else
> +		realfile = open_with_fake_path(&file->f_path, flags, realinode,
> +					       current_cred());
>  	revert_creds(old_cred);
>  
>  	pr_debug("open(%p[%pD2/%c], 0%o) -> (%p, 0%o)\n",
> 
> base-commit: 48f168950cd193d9beedc0afb0c79a254e32f637
> -- 
> 2.26.2
> 
> 
> -- 
> kernel-team mailing list
> kernel-team@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team
Kleber Souza April 24, 2020, 1:28 p.m.
On 23.04.20 12:11, Christian Brauner wrote:
> From: Andrei Vagin <avagin@gmail.com>
> 
> BugLink: https://bugs.launchpad.net/bugs/1857257
> 
> The hack was introduced in ("UBUNTU: SAUCE: overlayfs: allow with
> shiftfs as underlay") and it broke checkpoint/restore of docker
> contains:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1857257
> 
> The following script can be used to trigger the issue:
>   #!/bin/bash
> 
>   cat > test.py << EOF
>   import sys
> 
>   f = open("/proc/self/maps")
> 
>   for l in f.readlines():
>     if "python" not in l:
>       continue
>     print(l)
>     s = l.split()
>     start, end = s[0].split("-")
>     fname = s[-1]
>     print(start, end, fname)
>     break
>   else:
>     sys.exit(1)
> 
>   test_file1 = open(fname)
>   test_file2 = open("/proc/self/map_files/%s-%s" % (start, end))
> 
>   fdinfo1 = open("/proc/self/fdinfo/%d" % test_file1.fileno()).read()
>   fdinfo2 = open("/proc/self/fdinfo/%d" % test_file2.fileno()).read()
> 
>   if fdinfo1 != fdinfo2:
>     print("FAIL")
>     print(test_file1)
>     print(fdinfo1)
>     print(test_file2)
>     print(fdinfo2)
>     sys.exit(1)
>   print("PASS")
>   EOF
>   sudo docker run -it --privileged --rm -v `pwd`:/mnt python python /mnt/test.py
> 
> Cc: Stefan Bader <stefan.bader@canonical.com>
> Cc: Connor Kuehl <connor.kuehl@canonical.com>
> Cc: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
> Fixes: 58009298c6bd ("UBUNTU: SAUCE: overlayfs: allow with shiftfs as underlay")
> Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
> Signed-off-by: Andrei Vagin <avagin@gmail.com>
> ---
>  fs/overlayfs/file.c | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> index 4f94ef663569..9d16fff5342a 100644
> --- a/fs/overlayfs/file.c
> +++ b/fs/overlayfs/file.c
> @@ -42,9 +42,12 @@ static struct file *ovl_open_realfile(const struct file *file,
>  	int flags = file->f_flags | O_NOATIME | FMODE_NONOTIFY;
>  
>  	old_cred = ovl_override_creds(inode->i_sb);
> -	ovl_path_real(file->f_path.dentry, &realpath);
> -	realfile = open_with_fake_path(&realpath, flags, realinode,
> -				       current_cred());
> +	if (realpath.dentry->d_sb->s_magic == SHIFTFS_MAGIC)
> +		realfile = open_with_fake_path(&realpath, flags, realinode,
> +					       current_cred());
> +	else
> +		realfile = open_with_fake_path(&file->f_path, flags, realinode,
> +					       current_cred());
>  	revert_creds(old_cred);
>  
>  	pr_debug("open(%p[%pD2/%c], 0%o) -> (%p, 0%o)\n",
> 
> base-commit: 48f168950cd193d9beedc0afb0c79a254e32f637
> 

According to the replies on the other shiftfs threads, I have applied this patch
as well to all the stable series supporting shiftfs: eoan/linux and focal/linux.

Thanks,
Kleber
Khaled Elmously April 26, 2020, 11:02 p.m.
Looks like this patch breaks the Eoan build:

/tmp/kernel-kmously-e9d4a02-jZGj/build/fs/overlayfs/file.c: In function 'ovl_open_realfile':
/tmp/kernel-kmously-e9d4a02-jZGj/build/fs/overlayfs/file.c:34:40: error: 'SHIFTFS_MAGIC' undeclared (first use in this function)
   34 |  if (realpath.dentry->d_sb->s_magic == SHIFTFS_MAGIC)
      |                                        ^~~~~~~~~~~~~
/tmp/kernel-kmously-e9d4a02-jZGj/build/fs/overlayfs/file.c:34:40: note: each undeclared identifier is reported only once for each function it appears in




I haven't tried building focal 




On 2020-04-24 15:28:40 , Kleber Souza wrote:
> On 23.04.20 12:11, Christian Brauner wrote:
> > From: Andrei Vagin <avagin@gmail.com>
> > 
> > BugLink: https://bugs.launchpad.net/bugs/1857257
> > 
> > The hack was introduced in ("UBUNTU: SAUCE: overlayfs: allow with
> > shiftfs as underlay") and it broke checkpoint/restore of docker
> > contains:
> > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1857257
> > 
> > The following script can be used to trigger the issue:
> >   #!/bin/bash
> > 
> >   cat > test.py << EOF
> >   import sys
> > 
> >   f = open("/proc/self/maps")
> > 
> >   for l in f.readlines():
> >     if "python" not in l:
> >       continue
> >     print(l)
> >     s = l.split()
> >     start, end = s[0].split("-")
> >     fname = s[-1]
> >     print(start, end, fname)
> >     break
> >   else:
> >     sys.exit(1)
> > 
> >   test_file1 = open(fname)
> >   test_file2 = open("/proc/self/map_files/%s-%s" % (start, end))
> > 
> >   fdinfo1 = open("/proc/self/fdinfo/%d" % test_file1.fileno()).read()
> >   fdinfo2 = open("/proc/self/fdinfo/%d" % test_file2.fileno()).read()
> > 
> >   if fdinfo1 != fdinfo2:
> >     print("FAIL")
> >     print(test_file1)
> >     print(fdinfo1)
> >     print(test_file2)
> >     print(fdinfo2)
> >     sys.exit(1)
> >   print("PASS")
> >   EOF
> >   sudo docker run -it --privileged --rm -v `pwd`:/mnt python python /mnt/test.py
> > 
> > Cc: Stefan Bader <stefan.bader@canonical.com>
> > Cc: Connor Kuehl <connor.kuehl@canonical.com>
> > Cc: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
> > Fixes: 58009298c6bd ("UBUNTU: SAUCE: overlayfs: allow with shiftfs as underlay")
> > Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
> > Signed-off-by: Andrei Vagin <avagin@gmail.com>
> > ---
> >  fs/overlayfs/file.c | 9 ++++++---
> >  1 file changed, 6 insertions(+), 3 deletions(-)
> > 
> > diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
> > index 4f94ef663569..9d16fff5342a 100644
> > --- a/fs/overlayfs/file.c
> > +++ b/fs/overlayfs/file.c
> > @@ -42,9 +42,12 @@ static struct file *ovl_open_realfile(const struct file *file,
> >  	int flags = file->f_flags | O_NOATIME | FMODE_NONOTIFY;
> >  
> >  	old_cred = ovl_override_creds(inode->i_sb);
> > -	ovl_path_real(file->f_path.dentry, &realpath);
> > -	realfile = open_with_fake_path(&realpath, flags, realinode,
> > -				       current_cred());
> > +	if (realpath.dentry->d_sb->s_magic == SHIFTFS_MAGIC)
> > +		realfile = open_with_fake_path(&realpath, flags, realinode,
> > +					       current_cred());
> > +	else
> > +		realfile = open_with_fake_path(&file->f_path, flags, realinode,
> > +					       current_cred());
> >  	revert_creds(old_cred);
> >  
> >  	pr_debug("open(%p[%pD2/%c], 0%o) -> (%p, 0%o)\n",
> > 
> > base-commit: 48f168950cd193d9beedc0afb0c79a254e32f637
> > 
> 
> According to the replies on the other shiftfs threads, I have applied this patch
> as well to all the stable series supporting shiftfs: eoan/linux and focal/linux.
> 
> Thanks,
> Kleber
> 
> -- 
> kernel-team mailing list
> kernel-team@lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/kernel-team
Seth Forshee May 2, 2020, 2:12 a.m.
On Thu, Apr 23, 2020 at 12:11:28PM +0200, Christian Brauner wrote:
> From: Andrei Vagin <avagin@gmail.com>
> 
> BugLink: https://bugs.launchpad.net/bugs/1857257
> 
> The hack was introduced in ("UBUNTU: SAUCE: overlayfs: allow with
> shiftfs as underlay") and it broke checkpoint/restore of docker
> contains:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1857257
> 
> The following script can be used to trigger the issue:
>   #!/bin/bash
> 
>   cat > test.py << EOF
>   import sys
> 
>   f = open("/proc/self/maps")
> 
>   for l in f.readlines():
>     if "python" not in l:
>       continue
>     print(l)
>     s = l.split()
>     start, end = s[0].split("-")
>     fname = s[-1]
>     print(start, end, fname)
>     break
>   else:
>     sys.exit(1)
> 
>   test_file1 = open(fname)
>   test_file2 = open("/proc/self/map_files/%s-%s" % (start, end))
> 
>   fdinfo1 = open("/proc/self/fdinfo/%d" % test_file1.fileno()).read()
>   fdinfo2 = open("/proc/self/fdinfo/%d" % test_file2.fileno()).read()
> 
>   if fdinfo1 != fdinfo2:
>     print("FAIL")
>     print(test_file1)
>     print(fdinfo1)
>     print(test_file2)
>     print(fdinfo2)
>     sys.exit(1)
>   print("PASS")
>   EOF
>   sudo docker run -it --privileged --rm -v `pwd`:/mnt python python /mnt/test.py
> 
> Cc: Stefan Bader <stefan.bader@canonical.com>
> Cc: Connor Kuehl <connor.kuehl@canonical.com>
> Cc: Kleber Sacilotto de Souza <kleber.souza@canonical.com>
> Fixes: 58009298c6bd ("UBUNTU: SAUCE: overlayfs: allow with shiftfs as underlay")
> Acked-by: Christian Brauner <christian.brauner@ubuntu.com>
> Signed-off-by: Andrei Vagin <avagin@gmail.com>

Applied to unstable/master, with the uapi/linux/magic.h include added.
Thanks!